2024-04-18T09:51:56.152 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-18T09:51:56.152 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928 --description fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} --name teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi --owner scheduled_teuthology@teuthology --verbose -- /home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928/orig.config.yaml 2024-04-18T09:51:56.196 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T09:51:56.282 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928 branch: main description: fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} email: ceph-qa@ceph.com first_in_suite: false job_id: '7660928' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: a6549451253d26216d353246c0fe6f6d80f3b9cf ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a6549451253d26216d353246c0fe6f6d80f3b9cf thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 08d35a8d8529783882dd092c73c0b27be41c4d86 owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 7975 sha1: a6549451253d26216d353246c0fe6f6d80f3b9cf sleep_before_teardown: 0 subset: 475/512 suite: fs suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_08d35a8d8529783882dd092c73c0b27be41c4d86/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 08d35a8d8529783882dd092c73c0b27be41c4d86 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPPVfMD4DHC87XmWZt3TofmRlDjTt43T/RfV4ZFvgP2nfzE+gBYIP68S+VzT0UuVvprZV3CcxGQOBax7qpzfhkI= smithi145.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0dbhDmL1EjXSQwVknmV/Y3yOKu6mDKjuLciUf2po3nF29NJv8cXAwqEelT3Q3MjvvxUvBSEnsiWju2Ip+rIVs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_cephfs_shell teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-17_20:24:14 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-18T09:51:56.283 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_08d35a8d8529783882dd092c73c0b27be41c4d86/qa; will attempt to use it 2024-04-18T09:51:56.283 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_08d35a8d8529783882dd092c73c0b27be41c4d86/qa/tasks 2024-04-18T09:51:56.283 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-18T09:51:56.285 INFO:teuthology.task.internal:Checking packages... 2024-04-18T09:51:56.308 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'a6549451253d26216d353246c0fe6f6d80f3b9cf' 2024-04-18T09:51:56.308 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-18T09:51:56.308 INFO:teuthology.packaging:ref: None 2024-04-18T09:51:56.308 INFO:teuthology.packaging:tag: None 2024-04-18T09:51:56.308 INFO:teuthology.packaging:branch: main 2024-04-18T09:51:56.308 INFO:teuthology.packaging:sha1: a6549451253d26216d353246c0fe6f6d80f3b9cf 2024-04-18T09:51:56.308 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-04-18T09:51:56.616 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3141-g5be49cd9-1jammy 2024-04-18T09:51:56.618 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-18T09:51:56.625 INFO:teuthology.task.internal:no buildpackages task found 2024-04-18T09:51:56.625 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-18T09:51:56.661 INFO:teuthology.task.internal:Saving configuration 2024-04-18T09:51:56.677 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-18T09:51:56.685 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-18T09:51:56.711 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 09:46:24.217210', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYMB8bKtNCj93wcB/StgrfFIDDUZ4jY+9OOzBABMC0V'} 2024-04-18T09:51:56.732 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 09:46:24.215129', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIENnYqfG6r2M6xphg/ECt4GUTiQQMukGa1ZxGOVPQehu'} 2024-04-18T09:51:56.732 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-18T09:51:56.739 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-18T09:51:56.739 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-18T09:51:56.739 INFO:teuthology.run_tasks:Running task console_log... 2024-04-18T09:51:56.823 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f47126023a0>, signals=[15]) 2024-04-18T09:51:56.823 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-18T09:51:56.830 INFO:teuthology.task.internal:Opening connections... 2024-04-18T09:51:56.830 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T09:51:56.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T09:51:56.915 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2024-04-18T09:51:56.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T09:51:56.995 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-18T09:51:57.002 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2024-04-18T09:51:57.007 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2024-04-18T09:51:57.008 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-04-18T09:51:57.053 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-18T09:51:57.053 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="Ubuntu" 2024-04-18T09:51:57.053 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="22.04" 2024-04-18T09:51:57.053 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-18T09:51:57.054 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_CODENAME=jammy 2024-04-18T09:51:57.054 INFO:teuthology.orchestra.run.smithi062.stdout:ID=ubuntu 2024-04-18T09:51:57.054 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE=debian 2024-04-18T09:51:57.054 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-18T09:51:57.054 INFO:teuthology.orchestra.run.smithi062.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-18T09:51:57.054 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-18T09:51:57.054 INFO:teuthology.orchestra.run.smithi062.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-18T09:51:57.054 INFO:teuthology.orchestra.run.smithi062.stdout:UBUNTU_CODENAME=jammy 2024-04-18T09:51:57.055 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-04-18T09:51:57.073 DEBUG:teuthology.orchestra.run.smithi145:> uname -m 2024-04-18T09:51:57.078 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2024-04-18T09:51:57.079 DEBUG:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2024-04-18T09:51:57.124 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-18T09:51:57.124 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="Ubuntu" 2024-04-18T09:51:57.124 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="22.04" 2024-04-18T09:51:57.124 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-18T09:51:57.124 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_CODENAME=jammy 2024-04-18T09:51:57.124 INFO:teuthology.orchestra.run.smithi145.stdout:ID=ubuntu 2024-04-18T09:51:57.124 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE=debian 2024-04-18T09:51:57.124 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-18T09:51:57.124 INFO:teuthology.orchestra.run.smithi145.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-18T09:51:57.124 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-18T09:51:57.125 INFO:teuthology.orchestra.run.smithi145.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-18T09:51:57.125 INFO:teuthology.orchestra.run.smithi145.stdout:UBUNTU_CODENAME=jammy 2024-04-18T09:51:57.125 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2024-04-18T09:51:57.148 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-18T09:51:57.157 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-18T09:51:57.178 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-18T09:51:57.178 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2024-04-18T09:51:57.180 DEBUG:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2024-04-18T09:51:57.184 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-18T09:51:57.205 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-18T09:51:57.205 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2024-04-18T09:51:57.227 DEBUG:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2024-04-18T09:51:57.234 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-18T09:51:57.289 INFO:teuthology.run_tasks:Running task kernel... 2024-04-18T09:51:57.303 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-18T09:51:57.303 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-18T09:51:57.304 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-18T09:51:57.304 DEBUG:teuthology.orchestra.run.smithi145:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-18T09:51:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T09:51:57.309 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-04-18T09:51:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T09:51:57.310 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-04-18T09:51:57.357 INFO:teuthology.orchestra.run.smithi145.stdout:5.15.0-56-generic 2024-04-18T09:51:57.357 INFO:teuthology.task.kernel:Running kernel on smithi145: 5.15.0-56-generic 2024-04-18T09:51:57.357 DEBUG:teuthology.orchestra.run.smithi145:> sudo apt-get clean 2024-04-18T09:51:57.358 INFO:teuthology.orchestra.run.smithi062.stdout:5.15.0-56-generic 2024-04-18T09:51:57.358 INFO:teuthology.task.kernel:Running kernel on smithi062: 5.15.0-56-generic 2024-04-18T09:51:57.359 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get clean 2024-04-18T09:51:57.475 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get update 2024-04-18T09:51:57.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo apt-get update 2024-04-18T09:51:57.621 INFO:teuthology.orchestra.run.smithi062.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-18T09:51:57.629 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T09:51:57.631 INFO:teuthology.orchestra.run.smithi145.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-18T09:51:57.632 INFO:teuthology.orchestra.run.smithi145.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-18T09:51:57.721 INFO:teuthology.orchestra.run.smithi145.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-18T09:51:57.733 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T09:51:57.821 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-18T09:51:57.851 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,346 kB] 2024-04-18T09:51:57.858 INFO:teuthology.orchestra.run.smithi145.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,346 kB] 2024-04-18T09:51:57.926 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [611 kB] 2024-04-18T09:51:57.966 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-18T09:51:57.971 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [237 kB] 2024-04-18T09:51:57.973 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [445 kB] 2024-04-18T09:51:57.975 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-18T09:51:57.975 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [34.1 kB] 2024-04-18T09:51:57.975 INFO:teuthology.orchestra.run.smithi062.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,670 kB] 2024-04-18T09:51:57.981 INFO:teuthology.orchestra.run.smithi145.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [237 kB] 2024-04-18T09:51:57.986 INFO:teuthology.orchestra.run.smithi145.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-18T09:51:57.986 INFO:teuthology.orchestra.run.smithi145.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [34.1 kB] 2024-04-18T09:51:57.987 INFO:teuthology.orchestra.run.smithi145.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,670 kB] 2024-04-18T09:51:58.005 INFO:teuthology.orchestra.run.smithi062.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [280 kB] 2024-04-18T09:51:58.009 INFO:teuthology.orchestra.run.smithi062.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-18T09:51:58.010 INFO:teuthology.orchestra.run.smithi062.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-18T09:51:58.016 INFO:teuthology.orchestra.run.smithi145.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [280 kB] 2024-04-18T09:51:58.020 INFO:teuthology.orchestra.run.smithi062.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-18T09:51:58.022 INFO:teuthology.orchestra.run.smithi145.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-18T09:51:58.022 INFO:teuthology.orchestra.run.smithi145.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-04-18T09:51:58.024 INFO:teuthology.orchestra.run.smithi145.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,562 kB] 2024-04-18T09:51:58.038 INFO:teuthology.orchestra.run.smithi062.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-18T09:51:58.040 INFO:teuthology.orchestra.run.smithi062.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-18T09:51:58.041 INFO:teuthology.orchestra.run.smithi145.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [600 kB] 2024-04-18T09:51:58.043 INFO:teuthology.orchestra.run.smithi062.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-18T09:51:58.043 INFO:teuthology.orchestra.run.smithi062.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-18T09:51:58.044 INFO:teuthology.orchestra.run.smithi062.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-18T09:51:58.048 INFO:teuthology.orchestra.run.smithi145.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-18T09:51:58.050 INFO:teuthology.orchestra.run.smithi145.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-18T09:51:58.073 INFO:teuthology.orchestra.run.smithi145.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [298 kB] 2024-04-18T09:51:58.073 INFO:teuthology.orchestra.run.smithi145.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-18T09:51:58.073 INFO:teuthology.orchestra.run.smithi145.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-18T09:51:58.074 INFO:teuthology.orchestra.run.smithi145.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-18T09:51:58.074 INFO:teuthology.orchestra.run.smithi145.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-18T09:51:58.075 INFO:teuthology.orchestra.run.smithi145.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-18T09:51:58.076 INFO:teuthology.orchestra.run.smithi062.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-18T09:51:58.077 INFO:teuthology.orchestra.run.smithi145.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-18T09:51:58.078 INFO:teuthology.orchestra.run.smithi145.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,754 kB] 2024-04-18T09:51:58.099 INFO:teuthology.orchestra.run.smithi145.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [295 kB] 2024-04-18T09:51:58.101 INFO:teuthology.orchestra.run.smithi145.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-18T09:51:58.101 INFO:teuthology.orchestra.run.smithi145.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-18T09:51:58.108 INFO:teuthology.orchestra.run.smithi145.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,075 kB] 2024-04-18T09:51:58.124 INFO:teuthology.orchestra.run.smithi145.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-04-18T09:51:58.126 INFO:teuthology.orchestra.run.smithi145.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-18T09:51:58.126 INFO:teuthology.orchestra.run.smithi145.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-18T09:51:58.127 INFO:teuthology.orchestra.run.smithi145.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-18T09:51:58.127 INFO:teuthology.orchestra.run.smithi145.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-18T09:51:58.146 INFO:teuthology.orchestra.run.smithi145.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-18T09:51:58.146 INFO:teuthology.orchestra.run.smithi145.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-18T09:51:58.146 INFO:teuthology.orchestra.run.smithi145.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-18T09:51:58.147 INFO:teuthology.orchestra.run.smithi145.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-18T09:51:58.147 INFO:teuthology.orchestra.run.smithi145.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-18T09:51:58.165 INFO:teuthology.orchestra.run.smithi145.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-18T09:51:58.166 INFO:teuthology.orchestra.run.smithi145.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-18T09:51:58.167 INFO:teuthology.orchestra.run.smithi145.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-18T09:51:58.167 INFO:teuthology.orchestra.run.smithi145.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-18T09:51:58.183 INFO:teuthology.orchestra.run.smithi062.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-18T09:51:58.320 INFO:teuthology.orchestra.run.smithi062.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,562 kB] 2024-04-18T09:51:58.686 INFO:teuthology.orchestra.run.smithi062.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [611 kB] 2024-04-18T09:51:58.726 INFO:teuthology.orchestra.run.smithi062.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [298 kB] 2024-04-18T09:51:58.740 INFO:teuthology.orchestra.run.smithi062.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-18T09:51:58.741 INFO:teuthology.orchestra.run.smithi062.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-18T09:51:58.743 INFO:teuthology.orchestra.run.smithi062.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,754 kB] 2024-04-18T09:51:58.805 INFO:teuthology.orchestra.run.smithi062.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [295 kB] 2024-04-18T09:51:58.812 INFO:teuthology.orchestra.run.smithi062.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-18T09:51:58.978 INFO:teuthology.orchestra.run.smithi062.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [699 kB] 2024-04-18T09:51:59.567 INFO:teuthology.orchestra.run.smithi062.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,075 kB] 2024-04-18T09:51:59.683 INFO:teuthology.orchestra.run.smithi062.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-04-18T09:51:59.705 INFO:teuthology.orchestra.run.smithi062.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-18T09:51:59.709 INFO:teuthology.orchestra.run.smithi062.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-18T09:51:59.709 INFO:teuthology.orchestra.run.smithi062.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-18T09:51:59.710 INFO:teuthology.orchestra.run.smithi062.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-18T09:51:59.711 INFO:teuthology.orchestra.run.smithi062.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-18T09:51:59.711 INFO:teuthology.orchestra.run.smithi062.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.2 kB] 2024-04-18T09:51:59.716 INFO:teuthology.orchestra.run.smithi062.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-04-18T09:51:59.725 INFO:teuthology.orchestra.run.smithi062.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-18T09:51:59.893 INFO:teuthology.orchestra.run.smithi062.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-18T09:51:59.975 INFO:teuthology.orchestra.run.smithi062.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.4 kB] 2024-04-18T09:52:00.084 INFO:teuthology.orchestra.run.smithi062.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-04-18T09:52:00.141 INFO:teuthology.orchestra.run.smithi062.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-18T09:52:00.177 INFO:teuthology.orchestra.run.smithi062.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-18T09:52:03.641 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 12.9 MB in 2s (7,104 kB/s) 2024-04-18T09:52:04.749 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 12.9 MB in 3s (4,914 kB/s) 2024-04-18T09:52:04.978 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-04-18T09:52:04.995 DEBUG:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-18T09:52:05.046 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-04-18T09:52:05.222 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2024-04-18T09:52:05.223 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2024-04-18T09:52:05.375 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T09:52:05.376 INFO:teuthology.orchestra.run.smithi145.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T09:52:05.376 INFO:teuthology.orchestra.run.smithi145.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T09:52:05.376 INFO:teuthology.orchestra.run.smithi145.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T09:52:05.376 INFO:teuthology.orchestra.run.smithi145.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-18T09:52:05.376 INFO:teuthology.orchestra.run.smithi145.stdout: python2.7-minimal 2024-04-18T09:52:05.377 INFO:teuthology.orchestra.run.smithi145.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T09:52:05.377 INFO:teuthology.orchestra.run.smithi145.stdout:The following additional packages will be installed: 2024-04-18T09:52:05.378 INFO:teuthology.orchestra.run.smithi145.stdout: linux-generic linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic 2024-04-18T09:52:05.378 INFO:teuthology.orchestra.run.smithi145.stdout: linux-headers-generic linux-image-5.15.0-102-generic 2024-04-18T09:52:05.378 INFO:teuthology.orchestra.run.smithi145.stdout: linux-modules-5.15.0-102-generic linux-modules-extra-5.15.0-102-generic 2024-04-18T09:52:05.379 INFO:teuthology.orchestra.run.smithi145.stdout:Suggested packages: 2024-04-18T09:52:05.379 INFO:teuthology.orchestra.run.smithi145.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-18T09:52:05.379 INFO:teuthology.orchestra.run.smithi145.stdout:Recommended packages: 2024-04-18T09:52:05.379 INFO:teuthology.orchestra.run.smithi145.stdout: thermald 2024-04-18T09:52:05.419 INFO:teuthology.orchestra.run.smithi145.stdout:The following NEW packages will be installed: 2024-04-18T09:52:05.419 INFO:teuthology.orchestra.run.smithi145.stdout: linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic 2024-04-18T09:52:05.419 INFO:teuthology.orchestra.run.smithi145.stdout: linux-image-5.15.0-102-generic linux-modules-5.15.0-102-generic 2024-04-18T09:52:05.419 INFO:teuthology.orchestra.run.smithi145.stdout: linux-modules-extra-5.15.0-102-generic 2024-04-18T09:52:05.420 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages will be upgraded: 2024-04-18T09:52:05.421 INFO:teuthology.orchestra.run.smithi145.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-18T09:52:05.599 INFO:teuthology.orchestra.run.smithi145.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-18T09:52:05.600 INFO:teuthology.orchestra.run.smithi145.stdout:Need to get 113 MB of archives. 2024-04-18T09:52:05.600 INFO:teuthology.orchestra.run.smithi145.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-18T09:52:05.600 INFO:teuthology.orchestra.run.smithi145.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-102-generic amd64 5.15.0-102.112 [22.7 MB] 2024-04-18T09:52:06.110 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-18T09:52:06.127 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-18T09:52:06.191 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-18T09:52:06.394 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2024-04-18T09:52:06.394 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2024-04-18T09:52:06.573 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T09:52:06.574 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T09:52:06.574 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T09:52:06.574 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T09:52:06.574 INFO:teuthology.orchestra.run.smithi062.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-18T09:52:06.574 INFO:teuthology.orchestra.run.smithi062.stdout: python2.7-minimal 2024-04-18T09:52:06.575 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T09:52:06.575 INFO:teuthology.orchestra.run.smithi062.stdout:The following additional packages will be installed: 2024-04-18T09:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout: linux-generic linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic 2024-04-18T09:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout: linux-headers-generic linux-image-5.15.0-102-generic 2024-04-18T09:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout: linux-modules-5.15.0-102-generic linux-modules-extra-5.15.0-102-generic 2024-04-18T09:52:06.577 INFO:teuthology.orchestra.run.smithi062.stdout:Suggested packages: 2024-04-18T09:52:06.577 INFO:teuthology.orchestra.run.smithi062.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-18T09:52:06.577 INFO:teuthology.orchestra.run.smithi062.stdout:Recommended packages: 2024-04-18T09:52:06.578 INFO:teuthology.orchestra.run.smithi062.stdout: thermald 2024-04-18T09:52:06.629 INFO:teuthology.orchestra.run.smithi062.stdout:The following NEW packages will be installed: 2024-04-18T09:52:06.629 INFO:teuthology.orchestra.run.smithi062.stdout: linux-headers-5.15.0-102 linux-headers-5.15.0-102-generic 2024-04-18T09:52:06.629 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.15.0-102-generic linux-modules-5.15.0-102-generic 2024-04-18T09:52:06.629 INFO:teuthology.orchestra.run.smithi062.stdout: linux-modules-extra-5.15.0-102-generic 2024-04-18T09:52:06.631 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages will be upgraded: 2024-04-18T09:52:06.632 INFO:teuthology.orchestra.run.smithi062.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-18T09:52:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-18T09:52:06.698 INFO:teuthology.orchestra.run.smithi062.stdout:Need to get 113 MB of archives. 2024-04-18T09:52:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-18T09:52:06.699 INFO:teuthology.orchestra.run.smithi062.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-102-generic amd64 5.15.0-102.112 [22.7 MB] 2024-04-18T09:52:06.990 INFO:teuthology.orchestra.run.smithi145.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-102-generic amd64 5.15.0-102.112 [11.5 MB] 2024-04-18T09:52:07.291 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-102-generic amd64 5.15.0-102.112 [11.5 MB] 2024-04-18T09:52:07.415 INFO:teuthology.orchestra.run.smithi145.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-102-generic amd64 5.15.0-102.112 [63.9 MB] 2024-04-18T09:52:07.558 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-102-generic amd64 5.15.0-102.112 [63.9 MB] 2024-04-18T09:52:09.357 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.102.99 [1,700 B] 2024-04-18T09:52:09.367 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.102.99 [2,512 B] 2024-04-18T09:52:09.368 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-102 all 5.15.0-102.112 [12.4 MB] 2024-04-18T09:52:09.606 INFO:teuthology.orchestra.run.smithi145.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.102.99 [1,700 B] 2024-04-18T09:52:09.606 INFO:teuthology.orchestra.run.smithi145.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.102.99 [2,512 B] 2024-04-18T09:52:09.606 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-102 all 5.15.0-102.112 [12.4 MB] 2024-04-18T09:52:09.645 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-102-generic amd64 5.15.0-102.112 [2,877 kB] 2024-04-18T09:52:09.688 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.102.99 [2,360 B] 2024-04-18T09:52:10.117 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 113 MB in 3s (37.2 MB/s) 2024-04-18T09:52:10.275 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-modules-5.15.0-102-generic. 2024-04-18T09:52:10.673 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-102-generic amd64 5.15.0-102.112 [2,877 kB] 2024-04-18T09:52:10.754 INFO:teuthology.orchestra.run.smithi145.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.102.99 [2,360 B] 2024-04-18T09:52:11.165 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 113 MB in 5s (21.3 MB/s) 2024-04-18T09:52:11.326 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-modules-5.15.0-102-generic. 2024-04-18T09:52:12.244 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-18T09:52:12.248 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../0-linux-modules-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T09:52:12.322 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-modules-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:13.379 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-18T09:52:13.383 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../0-linux-modules-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T09:52:13.491 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-modules-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:14.482 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-image-5.15.0-102-generic. 2024-04-18T09:52:14.504 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../1-linux-image-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T09:52:14.642 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:15.102 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-102-generic. 2024-04-18T09:52:15.123 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T09:52:15.152 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-modules-extra-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:15.776 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-image-5.15.0-102-generic. 2024-04-18T09:52:15.797 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../1-linux-image-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T09:52:15.889 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:16.354 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-102-generic. 2024-04-18T09:52:16.374 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T09:52:16.575 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-modules-extra-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:21.988 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../3-linux-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T09:52:22.123 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T09:52:22.440 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T09:52:22.543 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-image-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T09:52:22.846 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-headers-5.15.0-102. 2024-04-18T09:52:22.870 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../5-linux-headers-5.15.0-102_5.15.0-102.112_all.deb ... 2024-04-18T09:52:22.913 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-headers-5.15.0-102 (5.15.0-102.112) ... 2024-04-18T09:52:23.659 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../3-linux-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T09:52:23.801 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T09:52:24.118 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T09:52:24.271 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-image-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T09:52:24.599 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-headers-5.15.0-102. 2024-04-18T09:52:24.622 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../5-linux-headers-5.15.0-102_5.15.0-102.112_all.deb ... 2024-04-18T09:52:24.666 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-headers-5.15.0-102 (5.15.0-102.112) ... 2024-04-18T09:52:28.871 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package linux-headers-5.15.0-102-generic. 2024-04-18T09:52:28.909 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../6-linux-headers-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T09:52:28.946 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-headers-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:30.379 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-headers-5.15.0-102-generic. 2024-04-18T09:52:30.416 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../6-linux-headers-5.15.0-102-generic_5.15.0-102.112_amd64.deb ... 2024-04-18T09:52:30.454 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-headers-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:31.717 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T09:52:31.842 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking linux-headers-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T09:52:32.176 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-headers-5.15.0-102 (5.15.0-102.112) ... 2024-04-18T09:52:32.314 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-headers-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:32.440 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-headers-generic (5.15.0.102.99) ... 2024-04-18T09:52:32.568 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:33.913 INFO:teuthology.orchestra.run.smithi062.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-18T09:52:33.913 INFO:teuthology.orchestra.run.smithi062.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-18T09:52:33.913 INFO:teuthology.orchestra.run.smithi062.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-102-generic 2024-04-18T09:52:33.913 INFO:teuthology.orchestra.run.smithi062.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-102-generic 2024-04-18T09:52:34.087 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-modules-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:34.518 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.102.99_amd64.deb ... 2024-04-18T09:52:34.660 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-headers-generic (5.15.0.102.99) over (5.15.0.56.54) ... 2024-04-18T09:52:35.010 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-headers-5.15.0-102 (5.15.0-102.112) ... 2024-04-18T09:52:35.115 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-headers-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:35.266 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-headers-generic (5.15.0.102.99) ... 2024-04-18T09:52:35.403 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:35.505 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-modules-extra-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:36.717 INFO:teuthology.orchestra.run.smithi145.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-18T09:52:36.717 INFO:teuthology.orchestra.run.smithi145.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-18T09:52:36.717 INFO:teuthology.orchestra.run.smithi145.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-102-generic 2024-04-18T09:52:36.717 INFO:teuthology.orchestra.run.smithi145.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-102-generic 2024-04-18T09:52:36.897 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-modules-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:36.923 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-image-generic (5.15.0.102.99) ... 2024-04-18T09:52:37.049 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up linux-generic (5.15.0.102.99) ... 2024-04-18T09:52:37.175 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:37.241 INFO:teuthology.orchestra.run.smithi062.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-18T09:52:37.241 INFO:teuthology.orchestra.run.smithi062.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-102-generic 2024-04-18T09:52:38.273 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-modules-extra-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:40.100 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-image-generic (5.15.0.102.99) ... 2024-04-18T09:52:40.226 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-generic (5.15.0.102.99) ... 2024-04-18T09:52:40.352 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for linux-image-5.15.0-102-generic (5.15.0-102.112) ... 2024-04-18T09:52:40.416 INFO:teuthology.orchestra.run.smithi145.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-18T09:52:40.416 INFO:teuthology.orchestra.run.smithi145.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-102-generic 2024-04-18T09:52:51.354 INFO:teuthology.orchestra.run.smithi062.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-18T09:52:51.354 INFO:teuthology.orchestra.run.smithi062.stdout:Sourcing file `/etc/default/grub' 2024-04-18T09:52:51.373 INFO:teuthology.orchestra.run.smithi062.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-18T09:52:51.389 INFO:teuthology.orchestra.run.smithi062.stdout:Generating grub configuration file ... 2024-04-18T09:52:51.673 INFO:teuthology.orchestra.run.smithi062.stdout:Found linux image: /boot/vmlinuz-5.15.0-102-generic 2024-04-18T09:52:51.685 INFO:teuthology.orchestra.run.smithi062.stdout:Found initrd image: /boot/initrd.img-5.15.0-102-generic 2024-04-18T09:52:52.053 INFO:teuthology.orchestra.run.smithi062.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-18T09:52:52.059 INFO:teuthology.orchestra.run.smithi062.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-18T09:52:52.123 INFO:teuthology.orchestra.run.smithi062.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-18T09:52:52.129 INFO:teuthology.orchestra.run.smithi062.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-18T09:52:52.300 INFO:teuthology.orchestra.run.smithi062.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-18T09:52:52.300 INFO:teuthology.orchestra.run.smithi062.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-18T09:52:52.300 INFO:teuthology.orchestra.run.smithi062.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-18T09:52:52.339 INFO:teuthology.orchestra.run.smithi062.stdout:done 2024-04-18T09:52:52.778 DEBUG:teuthology.orchestra.run.smithi062:> dpkg -s linux-image-generic 2024-04-18T09:52:52.803 INFO:teuthology.orchestra.run.smithi062.stdout:Package: linux-image-generic 2024-04-18T09:52:52.803 INFO:teuthology.orchestra.run.smithi062.stdout:Status: install ok installed 2024-04-18T09:52:52.803 INFO:teuthology.orchestra.run.smithi062.stdout:Priority: optional 2024-04-18T09:52:52.803 INFO:teuthology.orchestra.run.smithi062.stdout:Section: kernel 2024-04-18T09:52:52.803 INFO:teuthology.orchestra.run.smithi062.stdout:Installed-Size: 21 2024-04-18T09:52:52.803 INFO:teuthology.orchestra.run.smithi062.stdout:Maintainer: Ubuntu Kernel Team 2024-04-18T09:52:52.804 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: amd64 2024-04-18T09:52:52.804 INFO:teuthology.orchestra.run.smithi062.stdout:Source: linux-meta 2024-04-18T09:52:52.804 INFO:teuthology.orchestra.run.smithi062.stdout:Version: 5.15.0.102.99 2024-04-18T09:52:52.804 INFO:teuthology.orchestra.run.smithi062.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-102), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-18T09:52:52.804 INFO:teuthology.orchestra.run.smithi062.stdout:Depends: linux-image-5.15.0-102-generic, linux-modules-extra-5.15.0-102-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-18T09:52:52.804 INFO:teuthology.orchestra.run.smithi062.stdout:Recommends: thermald 2024-04-18T09:52:52.804 INFO:teuthology.orchestra.run.smithi062.stdout:Description: Generic Linux kernel image 2024-04-18T09:52:52.804 INFO:teuthology.orchestra.run.smithi062.stdout: This package will always depend on the latest generic kernel image 2024-04-18T09:52:52.805 INFO:teuthology.orchestra.run.smithi062.stdout: available. 2024-04-18T09:52:52.805 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-102-generic 2024-04-18T09:52:52.806 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-18T09:52:52.806 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-18T09:52:52.806 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-18T09:52:52.806 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=None, version=distro) 2024-04-18T09:52:52.806 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get clean 2024-04-18T09:52:53.022 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get update 2024-04-18T09:52:53.148 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T09:52:53.170 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-18T09:52:53.171 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-18T09:52:53.211 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-18T09:52:54.565 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-18T09:52:54.574 INFO:teuthology.orchestra.run.smithi145.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-18T09:52:54.574 INFO:teuthology.orchestra.run.smithi145.stdout:Sourcing file `/etc/default/grub' 2024-04-18T09:52:54.583 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-18T09:52:54.593 INFO:teuthology.orchestra.run.smithi145.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-18T09:52:54.615 INFO:teuthology.orchestra.run.smithi145.stdout:Generating grub configuration file ... 2024-04-18T09:52:54.635 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-18T09:52:54.818 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2024-04-18T09:52:54.818 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2024-04-18T09:52:54.937 INFO:teuthology.orchestra.run.smithi145.stdout:Found linux image: /boot/vmlinuz-5.15.0-102-generic 2024-04-18T09:52:54.957 INFO:teuthology.orchestra.run.smithi145.stdout:Found initrd image: /boot/initrd.img-5.15.0-102-generic 2024-04-18T09:52:54.980 INFO:teuthology.orchestra.run.smithi062.stdout:linux-image-generic is already the newest version (5.15.0.102.99). 2024-04-18T09:52:54.980 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T09:52:54.980 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T09:52:54.980 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T09:52:54.980 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T09:52:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-18T09:52:54.981 INFO:teuthology.orchestra.run.smithi062.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-18T09:52:54.981 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T09:52:55.022 INFO:teuthology.orchestra.run.smithi062.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-18T09:52:55.024 DEBUG:teuthology.orchestra.run.smithi062:> dpkg -s linux-image-generic 2024-04-18T09:52:55.084 INFO:teuthology.orchestra.run.smithi062.stdout:Package: linux-image-generic 2024-04-18T09:52:55.084 INFO:teuthology.orchestra.run.smithi062.stdout:Status: install ok installed 2024-04-18T09:52:55.084 INFO:teuthology.orchestra.run.smithi062.stdout:Priority: optional 2024-04-18T09:52:55.084 INFO:teuthology.orchestra.run.smithi062.stdout:Section: kernel 2024-04-18T09:52:55.084 INFO:teuthology.orchestra.run.smithi062.stdout:Installed-Size: 21 2024-04-18T09:52:55.084 INFO:teuthology.orchestra.run.smithi062.stdout:Maintainer: Ubuntu Kernel Team 2024-04-18T09:52:55.084 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: amd64 2024-04-18T09:52:55.084 INFO:teuthology.orchestra.run.smithi062.stdout:Source: linux-meta 2024-04-18T09:52:55.084 INFO:teuthology.orchestra.run.smithi062.stdout:Version: 5.15.0.102.99 2024-04-18T09:52:55.085 INFO:teuthology.orchestra.run.smithi062.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-102), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-18T09:52:55.085 INFO:teuthology.orchestra.run.smithi062.stdout:Depends: linux-image-5.15.0-102-generic, linux-modules-extra-5.15.0-102-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-18T09:52:55.085 INFO:teuthology.orchestra.run.smithi062.stdout:Recommends: thermald 2024-04-18T09:52:55.085 INFO:teuthology.orchestra.run.smithi062.stdout:Description: Generic Linux kernel image 2024-04-18T09:52:55.085 INFO:teuthology.orchestra.run.smithi062.stdout: This package will always depend on the latest generic kernel image 2024-04-18T09:52:55.085 INFO:teuthology.orchestra.run.smithi062.stdout: available. 2024-04-18T09:52:55.085 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-04-18T09:52:55.129 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.45HVefcAaE 2024-04-18T09:52:55.130 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub/grub.cfg /tmp/tmp.45HVefcAaE 2024-04-18T09:52:55.185 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.45HVefcAaE 2024-04-18T09:52:55.289 INFO:teuthology.orchestra.run.smithi145.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-18T09:52:55.294 INFO:teuthology.orchestra.run.smithi145.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-18T09:52:55.307 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.45HVefcAaE is 10KB 2024-04-18T09:52:55.321 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.45HVefcAaE 2024-04-18T09:52:55.326 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-18T09:52:55.370 INFO:teuthology.orchestra.run.smithi145.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-18T09:52:55.376 INFO:teuthology.orchestra.run.smithi145.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-18T09:52:55.381 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T09:52:55.381 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-18T09:52:55.381 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-18T09:52:55.448 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-102-generic 2024-04-18T09:52:55.448 DEBUG:teuthology.orchestra.run.smithi062:> sudo update-grub 2024-04-18T09:52:55.562 INFO:teuthology.orchestra.run.smithi145.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-18T09:52:55.562 INFO:teuthology.orchestra.run.smithi145.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-18T09:52:55.562 INFO:teuthology.orchestra.run.smithi145.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-18T09:52:55.608 INFO:teuthology.orchestra.run.smithi145.stdout:done 2024-04-18T09:52:56.056 DEBUG:teuthology.orchestra.run.smithi145:> dpkg -s linux-image-generic 2024-04-18T09:52:56.074 INFO:teuthology.orchestra.run.smithi145.stdout:Package: linux-image-generic 2024-04-18T09:52:56.074 INFO:teuthology.orchestra.run.smithi145.stdout:Status: install ok installed 2024-04-18T09:52:56.074 INFO:teuthology.orchestra.run.smithi145.stdout:Priority: optional 2024-04-18T09:52:56.074 INFO:teuthology.orchestra.run.smithi145.stdout:Section: kernel 2024-04-18T09:52:56.075 INFO:teuthology.orchestra.run.smithi145.stdout:Installed-Size: 21 2024-04-18T09:52:56.075 INFO:teuthology.orchestra.run.smithi145.stdout:Maintainer: Ubuntu Kernel Team 2024-04-18T09:52:56.075 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: amd64 2024-04-18T09:52:56.075 INFO:teuthology.orchestra.run.smithi145.stdout:Source: linux-meta 2024-04-18T09:52:56.075 INFO:teuthology.orchestra.run.smithi145.stdout:Version: 5.15.0.102.99 2024-04-18T09:52:56.075 INFO:teuthology.orchestra.run.smithi145.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-102), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-18T09:52:56.075 INFO:teuthology.orchestra.run.smithi145.stdout:Depends: linux-image-5.15.0-102-generic, linux-modules-extra-5.15.0-102-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-18T09:52:56.075 INFO:teuthology.orchestra.run.smithi145.stdout:Recommends: thermald 2024-04-18T09:52:56.076 INFO:teuthology.orchestra.run.smithi145.stdout:Description: Generic Linux kernel image 2024-04-18T09:52:56.076 INFO:teuthology.orchestra.run.smithi145.stdout: This package will always depend on the latest generic kernel image 2024-04-18T09:52:56.076 INFO:teuthology.orchestra.run.smithi145.stdout: available. 2024-04-18T09:52:56.076 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-102-generic 2024-04-18T09:52:56.076 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-18T09:52:56.076 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-18T09:52:56.077 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-18T09:52:56.077 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi145.front.sepia.ceph.com, path=None, version=distro) 2024-04-18T09:52:56.077 DEBUG:teuthology.orchestra.run.smithi145:> sudo apt-get clean 2024-04-18T09:52:56.302 DEBUG:teuthology.orchestra.run.smithi145:> sudo apt-get update 2024-04-18T09:52:56.565 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T09:52:56.565 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-18T09:52:56.626 INFO:teuthology.orchestra.run.smithi062.stderr:Sourcing file `/etc/default/grub' 2024-04-18T09:52:56.627 INFO:teuthology.orchestra.run.smithi062.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-18T09:52:56.631 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2024-04-18T09:52:56.636 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-18T09:52:56.718 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-18T09:52:56.805 INFO:teuthology.orchestra.run.smithi062.stderr:Found linux image: /boot/vmlinuz-5.15.0-102-generic 2024-04-18T09:52:56.811 INFO:teuthology.orchestra.run.smithi062.stderr:Found initrd image: /boot/initrd.img-5.15.0-102-generic 2024-04-18T09:52:57.029 INFO:teuthology.orchestra.run.smithi062.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-18T09:52:57.034 INFO:teuthology.orchestra.run.smithi062.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-18T09:52:57.071 INFO:teuthology.orchestra.run.smithi062.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-18T09:52:57.076 INFO:teuthology.orchestra.run.smithi062.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-18T09:52:57.189 INFO:teuthology.orchestra.run.smithi062.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-18T09:52:57.189 INFO:teuthology.orchestra.run.smithi062.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-18T09:52:57.189 INFO:teuthology.orchestra.run.smithi062.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-18T09:52:57.212 INFO:teuthology.orchestra.run.smithi062.stderr:done 2024-04-18T09:52:57.220 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2024-04-18T09:52:57.966 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-04-18T09:52:57.984 DEBUG:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-18T09:52:58.034 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-04-18T09:52:58.218 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2024-04-18T09:52:58.219 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2024-04-18T09:52:58.371 INFO:teuthology.orchestra.run.smithi145.stdout:linux-image-generic is already the newest version (5.15.0.102.99). 2024-04-18T09:52:58.372 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T09:52:58.372 INFO:teuthology.orchestra.run.smithi145.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T09:52:58.372 INFO:teuthology.orchestra.run.smithi145.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T09:52:58.372 INFO:teuthology.orchestra.run.smithi145.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T09:52:58.372 INFO:teuthology.orchestra.run.smithi145.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-18T09:52:58.372 INFO:teuthology.orchestra.run.smithi145.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-18T09:52:58.372 INFO:teuthology.orchestra.run.smithi145.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T09:52:58.412 INFO:teuthology.orchestra.run.smithi145.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-18T09:52:58.414 DEBUG:teuthology.orchestra.run.smithi145:> dpkg -s linux-image-generic 2024-04-18T09:52:58.467 INFO:teuthology.orchestra.run.smithi145.stdout:Package: linux-image-generic 2024-04-18T09:52:58.467 INFO:teuthology.orchestra.run.smithi145.stdout:Status: install ok installed 2024-04-18T09:52:58.467 INFO:teuthology.orchestra.run.smithi145.stdout:Priority: optional 2024-04-18T09:52:58.467 INFO:teuthology.orchestra.run.smithi145.stdout:Section: kernel 2024-04-18T09:52:58.467 INFO:teuthology.orchestra.run.smithi145.stdout:Installed-Size: 21 2024-04-18T09:52:58.467 INFO:teuthology.orchestra.run.smithi145.stdout:Maintainer: Ubuntu Kernel Team 2024-04-18T09:52:58.468 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: amd64 2024-04-18T09:52:58.468 INFO:teuthology.orchestra.run.smithi145.stdout:Source: linux-meta 2024-04-18T09:52:58.468 INFO:teuthology.orchestra.run.smithi145.stdout:Version: 5.15.0.102.99 2024-04-18T09:52:58.468 INFO:teuthology.orchestra.run.smithi145.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-102), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-18T09:52:58.468 INFO:teuthology.orchestra.run.smithi145.stdout:Depends: linux-image-5.15.0-102-generic, linux-modules-extra-5.15.0-102-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-18T09:52:58.468 INFO:teuthology.orchestra.run.smithi145.stdout:Recommends: thermald 2024-04-18T09:52:58.468 INFO:teuthology.orchestra.run.smithi145.stdout:Description: Generic Linux kernel image 2024-04-18T09:52:58.468 INFO:teuthology.orchestra.run.smithi145.stdout: This package will always depend on the latest generic kernel image 2024-04-18T09:52:58.468 INFO:teuthology.orchestra.run.smithi145.stdout: available. 2024-04-18T09:52:58.468 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-04-18T09:52:58.515 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.Kprpie6HUX 2024-04-18T09:52:58.515 DEBUG:teuthology.orchestra.run.smithi145:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Kprpie6HUX 2024-04-18T09:52:58.568 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.Kprpie6HUX 2024-04-18T09:52:58.685 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.Kprpie6HUX is 10KB 2024-04-18T09:52:58.698 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.Kprpie6HUX 2024-04-18T09:52:58.704 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-18T09:52:58.760 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T09:52:58.760 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-18T09:52:58.760 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-18T09:52:58.827 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-102-generic 2024-04-18T09:52:58.827 DEBUG:teuthology.orchestra.run.smithi145:> sudo update-grub 2024-04-18T09:53:00.131 INFO:teuthology.orchestra.run.smithi145.stderr:Sourcing file `/etc/default/grub' 2024-04-18T09:53:00.132 INFO:teuthology.orchestra.run.smithi145.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-18T09:53:00.135 INFO:teuthology.orchestra.run.smithi145.stderr:Generating grub configuration file ... 2024-04-18T09:53:00.306 INFO:teuthology.orchestra.run.smithi145.stderr:Found linux image: /boot/vmlinuz-5.15.0-102-generic 2024-04-18T09:53:00.312 INFO:teuthology.orchestra.run.smithi145.stderr:Found initrd image: /boot/initrd.img-5.15.0-102-generic 2024-04-18T09:53:00.589 INFO:teuthology.orchestra.run.smithi145.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-18T09:53:00.594 INFO:teuthology.orchestra.run.smithi145.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-18T09:53:00.635 INFO:teuthology.orchestra.run.smithi145.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-18T09:53:00.640 INFO:teuthology.orchestra.run.smithi145.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-18T09:53:00.751 INFO:teuthology.orchestra.run.smithi145.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-18T09:53:00.751 INFO:teuthology.orchestra.run.smithi145.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-18T09:53:00.752 INFO:teuthology.orchestra.run.smithi145.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-18T09:53:00.761 INFO:teuthology.orchestra.run.smithi145.stderr:done 2024-04-18T09:53:00.763 DEBUG:teuthology.orchestra.run.smithi145:> sudo shutdown -r now 2024-04-18T09:53:27.228 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-18T09:53:27.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-18T09:53:27.229 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T09:53:30.766 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-18T09:53:30.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-04-18T09:53:30.767 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T09:54:01.887 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-04-18T09:54:05.214 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-04-18T09:54:10.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-18T09:54:10.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T09:54:13.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-04-18T09:54:14.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-04-18T09:54:14.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T09:54:25.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-18T09:54:25.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T09:54:29.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-04-18T09:54:32.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-04-18T09:54:44.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-18T09:54:44.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T09:54:44.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-04-18T09:54:44.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T09:54:44.629 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-04-18T09:54:45.258 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-04-18T09:54:45.576 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-18T09:54:45.576 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-102-generic"... 2024-04-18T09:54:45.576 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-04-18T09:54:45.625 INFO:teuthology.orchestra.run.smithi062.stdout:5.15.0-102-generic 2024-04-18T09:54:45.625 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-102-generic vs 5.15.0-102-generic 2024-04-18T09:54:45.625 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-18T09:54:45.625 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-18T09:54:46.027 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-04-18T09:54:46.027 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-102-generic"... 2024-04-18T09:54:46.027 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-04-18T09:54:46.075 INFO:teuthology.orchestra.run.smithi145.stdout:5.15.0-102-generic 2024-04-18T09:54:46.075 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-102-generic vs 5.15.0-102-generic 2024-04-18T09:54:46.075 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-18T09:54:46.075 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-18T09:54:46.626 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-18T09:54:46.626 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-18T09:54:47.075 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-18T09:54:47.076 DEBUG:teuthology.orchestra.run.smithi145:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-18T09:54:47.144 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2024-04-18T09:54:47.158 DEBUG:teuthology.parallel:result is None 2024-04-18T09:54:47.196 INFO:teuthology.orchestra.run.smithi145.stdout:ttyS1 2024-04-18T09:54:47.229 DEBUG:teuthology.parallel:result is None 2024-04-18T09:54:47.229 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-18T09:54:47.236 INFO:teuthology.task.internal:Creating test directory... 2024-04-18T09:54:47.236 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-18T09:54:47.238 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-18T09:54:47.243 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-18T09:54:47.260 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-18T09:54:47.294 INFO:teuthology.task.internal:Creating archive directory... 2024-04-18T09:54:47.294 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-18T09:54:47.296 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-18T09:54:47.315 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-18T09:54:47.327 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-18T09:54:47.328 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-18T09:54:47.354 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-18T09:54:47.371 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-18T09:54:47.379 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-18T09:54:47.390 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-18T09:54:47.398 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-18T09:54:47.400 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-18T09:54:47.407 INFO:teuthology.task.internal:Configuring sudo... 2024-04-18T09:54:47.407 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-18T09:54:47.430 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-18T09:54:47.453 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-18T09:54:47.462 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-18T09:54:47.462 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-18T09:54:47.486 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-18T09:54:47.498 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-18T09:54:47.537 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-18T09:54:47.585 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T09:54:47.585 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-18T09:54:47.643 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-18T09:54:47.649 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-18T09:54:47.695 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T09:54:47.695 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-18T09:54:47.761 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2024-04-18T09:54:47.763 DEBUG:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2024-04-18T09:54:47.854 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-18T09:54:47.862 INFO:teuthology.task.internal:Starting timer... 2024-04-18T09:54:47.862 INFO:teuthology.run_tasks:Running task pcp... 2024-04-18T09:54:47.940 INFO:teuthology.run_tasks:Running task selinux... 2024-04-18T09:54:47.972 DEBUG:teuthology.task.selinux:Excluding smithi062: OS 'ubuntu' does not support SELinux 2024-04-18T09:54:47.972 DEBUG:teuthology.task.selinux:Excluding smithi145: OS 'ubuntu' does not support SELinux 2024-04-18T09:54:47.973 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-18T09:54:47.973 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-18T09:54:47.973 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-18T09:54:47.973 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-18T09:54:48.005 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-18T09:54:48.023 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-18T09:54:48.025 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi062.front.sepia.ceph.com,smithi145.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-18T10:01:58.381 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi062.front.sepia.ceph.com'), Remote(name='ubuntu@smithi145.front.sepia.ceph.com')] 2024-04-18T10:01:58.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-18T10:01:58.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T10:01:58.468 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-04-18T10:01:58.554 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-04-18T10:01:58.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-04-18T10:01:58.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-18T10:01:58.638 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-04-18T10:01:58.718 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-04-18T10:01:58.718 INFO:teuthology.run_tasks:Running task clock... 2024-04-18T10:01:58.729 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-18T10:01:58.729 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-18T10:01:58.729 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-18T10:01:58.732 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-18T10:01:58.732 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-18T10:01:58.760 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-18T10:01:58.760 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: Command line: ntpd -gq 2024-04-18T10:01:58.760 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: ---------------------------------------------------- 2024-04-18T10:01:58.760 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: ntp-4 is maintained by Network Time Foundation, 2024-04-18T10:01:58.760 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-18T10:01:58.761 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: corporation. Support and training for ntp-4 are 2024-04-18T10:01:58.761 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: available at https://www.nwtime.org/support 2024-04-18T10:01:58.761 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: ---------------------------------------------------- 2024-04-18T10:01:58.761 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: proto: precision = 0.066 usec (-24) 2024-04-18T10:01:58.761 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: basedate set to 2022-02-04 2024-04-18T10:01:58.762 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: gps base set to 2022-02-06 (week 2196) 2024-04-18T10:01:58.762 INFO:teuthology.orchestra.run.smithi062.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-18T10:01:58.762 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-18T10:01:58.762 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: restrict ::: KOD does nothing without LIMITED. 2024-04-18T10:01:58.763 INFO:teuthology.orchestra.run.smithi062.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-18T10:01:58.763 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: Listen and drop on 0 v6wildcard [::]:123 2024-04-18T10:01:58.764 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-18T10:01:58.764 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: Listen normally on 2 lo 127.0.0.1:123 2024-04-18T10:01:58.764 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: Listen normally on 3 enp3s0f1 172.21.15.62:123 2024-04-18T10:01:58.764 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: Listen normally on 4 lo [::1]:123 2024-04-18T10:01:58.764 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:febb:d1c3%5]:123 2024-04-18T10:01:58.764 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:01:58 ntpd[17183]: Listening on routing socket on fd #22 for interface updates 2024-04-18T10:01:58.793 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-18T10:01:58.793 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: Command line: ntpd -gq 2024-04-18T10:01:58.793 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: ---------------------------------------------------- 2024-04-18T10:01:58.793 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: ntp-4 is maintained by Network Time Foundation, 2024-04-18T10:01:58.794 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-18T10:01:58.794 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: corporation. Support and training for ntp-4 are 2024-04-18T10:01:58.794 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: available at https://www.nwtime.org/support 2024-04-18T10:01:58.794 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: ---------------------------------------------------- 2024-04-18T10:01:58.794 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: proto: precision = 0.060 usec (-24) 2024-04-18T10:01:58.795 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: basedate set to 2022-02-04 2024-04-18T10:01:58.795 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: gps base set to 2022-02-06 (week 2196) 2024-04-18T10:01:58.795 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-18T10:01:58.795 INFO:teuthology.orchestra.run.smithi145.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-18T10:01:58.795 INFO:teuthology.orchestra.run.smithi145.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-18T10:01:58.796 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: restrict ::: KOD does nothing without LIMITED. 2024-04-18T10:01:58.796 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: Listen and drop on 0 v6wildcard [::]:123 2024-04-18T10:01:58.796 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-18T10:01:58.796 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: Listen normally on 2 lo 127.0.0.1:123 2024-04-18T10:01:58.796 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: Listen normally on 3 enp3s0f1 172.21.15.145:123 2024-04-18T10:01:58.797 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: Listen normally on 4 lo [::1]:123 2024-04-18T10:01:58.797 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7791%5]:123 2024-04-18T10:01:58.797 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:01:58 ntpd[17213]: Listening on routing socket on fd #22 for interface updates 2024-04-18T10:01:59.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:01:59.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:01:59.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:01:59.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:01:59.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:01:59.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:01:59.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:01:59.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:01:59.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:01:59.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:01:59.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:01:59.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:01:59.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:01:59.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:01:59 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:00.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:00 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:00.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:00 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:00.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:00 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:00.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:00 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:00.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:00 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:00.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:00 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:01.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:01 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:01.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:01 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:01.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:01 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:01.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:01 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:01.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:01 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:01.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:01 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:01.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:01 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:01.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:01 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:01.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:01 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:01.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:01 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:01.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:01 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:01.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:01 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:02.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:02 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:02.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:02 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:02.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:02 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:02.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:02 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:02.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:02 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:02.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:02 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:02.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:02.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:02.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:02.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:02.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:02.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:02.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:02.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:02.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:02.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:02.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:02.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:02.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:02.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:02.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:02.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:02.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:02.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:02 ntpd[17213]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:03 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:03 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:03 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:03 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:03.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:03 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:03 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:03 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:03 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:03 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:03 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:03 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:03.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:03 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:03.794 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:03 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:03.794 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:03 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:04.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:04 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:04.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:04 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:04.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:04 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:04.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:04 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:04.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:04 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:04.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:04 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:04.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:04.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:04.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:04.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:04.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:04.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:04.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:04.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:04.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:04.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:04.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:04.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:04.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:04.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:04.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:04.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:04.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:04.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:04 ntpd[17213]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:05.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:05 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:05.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:05 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:05.761 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:05 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:05.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:05 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:05.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:05 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:05.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:05 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:05.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:05 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:05.762 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:05 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:05.762 INFO:teuthology.orchestra.run.smithi062.stdout:18 Apr 10:02:05 ntpd[17183]: ntpd: time slew -0.000154 s 2024-04-18T10:02:05.762 INFO:teuthology.orchestra.run.smithi062.stdout:ntpd: time slew -0.000154s 2024-04-18T10:02:05.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:05 ntpd[17183]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-18T10:02:05.763 INFO:teuthology.orchestra.run.smithi062.stderr:18 Apr 10:02:05 ntpd[17183]: can't open /var/log/ntpstats/loopstats.20240418: Permission denied 2024-04-18T10:02:05.814 INFO:teuthology.orchestra.run.smithi062.stdout: remote refid st t when poll reach delay offset jitter 2024-04-18T10:02:05.814 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================== 2024-04-18T10:02:05.814 INFO:teuthology.orchestra.run.smithi062.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T10:02:05.814 INFO:teuthology.orchestra.run.smithi062.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T10:02:05.814 INFO:teuthology.orchestra.run.smithi062.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T10:02:05.815 INFO:teuthology.orchestra.run.smithi062.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T10:02:06.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:06.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:06.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:06.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:06.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:06.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:06.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:06.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:06.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:06.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:06.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:06.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:06.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:06.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:06.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:06.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:06.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:06.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:06 ntpd[17213]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:08.794 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:08 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:08.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:08 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:08.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:08 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:08.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:08 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:08.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:08 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:08.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:08 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:08.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:08 ntpd[17213]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-18T10:02:08.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:08 ntpd[17213]: can't open /var/log/ntpstats/rawstats.20240418: Permission denied 2024-04-18T10:02:08.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:08 ntpd[17213]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-18T10:02:08.795 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:08 ntpd[17213]: can't open /var/log/ntpstats/peerstats.20240418: Permission denied 2024-04-18T10:02:08.795 INFO:teuthology.orchestra.run.smithi145.stdout:18 Apr 10:02:08 ntpd[17213]: ntpd: time slew -0.001493 s 2024-04-18T10:02:08.796 INFO:teuthology.orchestra.run.smithi145.stdout:ntpd: time slew -0.001493s 2024-04-18T10:02:08.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:08 ntpd[17213]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-18T10:02:08.796 INFO:teuthology.orchestra.run.smithi145.stderr:18 Apr 10:02:08 ntpd[17213]: can't open /var/log/ntpstats/loopstats.20240418: Permission denied 2024-04-18T10:02:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: remote refid st t when poll reach delay offset jitter 2024-04-18T10:02:08.837 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================== 2024-04-18T10:02:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T10:02:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T10:02:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T10:02:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T10:02:08.838 INFO:teuthology.run_tasks:Running task install... 2024-04-18T10:02:08.847 DEBUG:teuthology.task.install:project ceph 2024-04-18T10:02:08.847 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a6549451253d26216d353246c0fe6f6d80f3b9cf'}} 2024-04-18T10:02:08.847 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a6549451253d26216d353246c0fe6f6d80f3b9cf'} 2024-04-18T10:02:08.847 INFO:teuthology.task.install:Using flavor: default 2024-04-18T10:02:08.853 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-18T10:02:08.853 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-18T10:02:08.854 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-key list | grep Ceph 2024-04-18T10:02:08.854 DEBUG:teuthology.orchestra.run.smithi145:> sudo apt-key list | grep Ceph 2024-04-18T10:02:08.922 INFO:teuthology.orchestra.run.smithi062.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-18T10:02:08.936 INFO:teuthology.orchestra.run.smithi145.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-18T10:02:08.954 INFO:teuthology.orchestra.run.smithi062.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-18T10:02:08.954 INFO:teuthology.orchestra.run.smithi062.stdout:uid [ unknown] Ceph.com (release key) 2024-04-18T10:02:08.955 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-18T10:02:08.955 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-18T10:02:08.956 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a6549451253d26216d353246c0fe6f6d80f3b9cf 2024-04-18T10:02:08.960 INFO:teuthology.orchestra.run.smithi145.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-18T10:02:08.960 INFO:teuthology.orchestra.run.smithi145.stdout:uid [ unknown] Ceph.com (release key) 2024-04-18T10:02:08.960 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-18T10:02:08.960 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-18T10:02:08.960 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a6549451253d26216d353246c0fe6f6d80f3b9cf 2024-04-18T10:02:09.102 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default/ 2024-04-18T10:02:09.102 INFO:teuthology.task.install.deb:Package version is 19.0.0-3115-ga6549451-1jammy 2024-04-18T10:02:09.171 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default/ 2024-04-18T10:02:09.172 INFO:teuthology.task.install.deb:Package version is 19.0.0-3115-ga6549451-1jammy 2024-04-18T10:02:09.492 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:02:09.493 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-18T10:02:09.507 DEBUG:teuthology.orchestra.run.smithi145:> sudo apt-get update 2024-04-18T10:02:09.565 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:02:09.566 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-18T10:02:09.580 DEBUG:teuthology.orchestra.run.smithi062:> sudo apt-get update 2024-04-18T10:02:09.756 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-18T10:02:09.756 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T10:02:09.771 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-18T10:02:09.792 INFO:teuthology.orchestra.run.smithi062.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-18T10:02:09.805 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-18T10:02:09.805 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-18T10:02:09.878 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-18T10:02:09.956 INFO:teuthology.orchestra.run.smithi145.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy InRelease 2024-04-18T10:02:09.960 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-18T10:02:10.033 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-18T10:02:10.094 INFO:teuthology.orchestra.run.smithi062.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy InRelease 2024-04-18T10:02:10.112 INFO:teuthology.orchestra.run.smithi145.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-18T10:02:10.190 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-18T10:02:10.192 INFO:teuthology.orchestra.run.smithi145.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-18T10:02:10.283 INFO:teuthology.orchestra.run.smithi062.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-18T10:02:10.357 INFO:teuthology.orchestra.run.smithi145.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2024-04-18T10:02:10.377 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,417 B] 2024-04-18T10:02:10.439 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 29.2 kB in 1s (39.3 kB/s) 2024-04-18T10:02:10.481 INFO:teuthology.orchestra.run.smithi062.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-18T10:02:10.579 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 29.2 kB in 1s (36.7 kB/s) 2024-04-18T10:02:11.373 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-04-18T10:02:11.389 DEBUG:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3115-ga6549451-1jammy cephadm=19.0.0-3115-ga6549451-1jammy ceph-mds=19.0.0-3115-ga6549451-1jammy ceph-mgr=19.0.0-3115-ga6549451-1jammy ceph-common=19.0.0-3115-ga6549451-1jammy ceph-fuse=19.0.0-3115-ga6549451-1jammy ceph-test=19.0.0-3115-ga6549451-1jammy ceph-volume=19.0.0-3115-ga6549451-1jammy radosgw=19.0.0-3115-ga6549451-1jammy python3-rados=19.0.0-3115-ga6549451-1jammy python3-rgw=19.0.0-3115-ga6549451-1jammy python3-cephfs=19.0.0-3115-ga6549451-1jammy python3-rbd=19.0.0-3115-ga6549451-1jammy libcephfs2=19.0.0-3115-ga6549451-1jammy libcephfs-dev=19.0.0-3115-ga6549451-1jammy librados2=19.0.0-3115-ga6549451-1jammy librbd1=19.0.0-3115-ga6549451-1jammy rbd-fuse=19.0.0-3115-ga6549451-1jammy python3-cephfs=19.0.0-3115-ga6549451-1jammy cephfs-shell=19.0.0-3115-ga6549451-1jammy cephfs-top=19.0.0-3115-ga6549451-1jammy cephfs-mirror=19.0.0-3115-ga6549451-1jammy 2024-04-18T10:02:11.448 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-04-18T10:02:11.574 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-18T10:02:11.589 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3115-ga6549451-1jammy cephadm=19.0.0-3115-ga6549451-1jammy ceph-mds=19.0.0-3115-ga6549451-1jammy ceph-mgr=19.0.0-3115-ga6549451-1jammy ceph-common=19.0.0-3115-ga6549451-1jammy ceph-fuse=19.0.0-3115-ga6549451-1jammy ceph-test=19.0.0-3115-ga6549451-1jammy ceph-volume=19.0.0-3115-ga6549451-1jammy radosgw=19.0.0-3115-ga6549451-1jammy python3-rados=19.0.0-3115-ga6549451-1jammy python3-rgw=19.0.0-3115-ga6549451-1jammy python3-cephfs=19.0.0-3115-ga6549451-1jammy python3-rbd=19.0.0-3115-ga6549451-1jammy libcephfs2=19.0.0-3115-ga6549451-1jammy libcephfs-dev=19.0.0-3115-ga6549451-1jammy librados2=19.0.0-3115-ga6549451-1jammy librbd1=19.0.0-3115-ga6549451-1jammy rbd-fuse=19.0.0-3115-ga6549451-1jammy python3-cephfs=19.0.0-3115-ga6549451-1jammy cephfs-shell=19.0.0-3115-ga6549451-1jammy cephfs-top=19.0.0-3115-ga6549451-1jammy cephfs-mirror=19.0.0-3115-ga6549451-1jammy 2024-04-18T10:02:11.629 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2024-04-18T10:02:11.629 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2024-04-18T10:02:11.655 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-18T10:02:11.801 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T10:02:11.801 INFO:teuthology.orchestra.run.smithi145.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T10:02:11.801 INFO:teuthology.orchestra.run.smithi145.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T10:02:11.801 INFO:teuthology.orchestra.run.smithi145.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T10:02:11.801 INFO:teuthology.orchestra.run.smithi145.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-18T10:02:11.802 INFO:teuthology.orchestra.run.smithi145.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-18T10:02:11.802 INFO:teuthology.orchestra.run.smithi145.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T10:02:11.802 INFO:teuthology.orchestra.run.smithi145.stdout:The following additional packages will be installed: 2024-04-18T10:02:11.802 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-18T10:02:11.802 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-18T10:02:11.802 INFO:teuthology.orchestra.run.smithi145.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-18T10:02:11.803 INFO:teuthology.orchestra.run.smithi145.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-18T10:02:11.803 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-18T10:02:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-18T10:02:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-18T10:02:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-18T10:02:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-18T10:02:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-18T10:02:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-18T10:02:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-18T10:02:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-18T10:02:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-18T10:02:11.804 INFO:teuthology.orchestra.run.smithi145.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-18T10:02:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-18T10:02:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-18T10:02:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-18T10:02:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-18T10:02:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-18T10:02:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-18T10:02:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-18T10:02:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-18T10:02:11.805 INFO:teuthology.orchestra.run.smithi145.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-18T10:02:11.805 INFO:teuthology.orchestra.run.smithi145.stdout:Suggested packages: 2024-04-18T10:02:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-18T10:02:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-18T10:02:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-18T10:02:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-18T10:02:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-18T10:02:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-18T10:02:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-18T10:02:11.806 INFO:teuthology.orchestra.run.smithi145.stdout:Recommended packages: 2024-04-18T10:02:11.806 INFO:teuthology.orchestra.run.smithi145.stdout: btrfs-tools 2024-04-18T10:02:11.853 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2024-04-18T10:02:11.854 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2024-04-18T10:02:11.910 INFO:teuthology.orchestra.run.smithi145.stdout:The following NEW packages will be installed: 2024-04-18T10:02:11.910 INFO:teuthology.orchestra.run.smithi145.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-18T10:02:11.910 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-18T10:02:11.910 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-18T10:02:11.910 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-18T10:02:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-18T10:02:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-18T10:02:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-18T10:02:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-18T10:02:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-18T10:02:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-18T10:02:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-18T10:02:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-18T10:02:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-18T10:02:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-18T10:02:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-18T10:02:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-18T10:02:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-18T10:02:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-18T10:02:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-18T10:02:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-18T10:02:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-18T10:02:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-18T10:02:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-18T10:02:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-18T10:02:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-18T10:02:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-18T10:02:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-18T10:02:11.913 INFO:teuthology.orchestra.run.smithi145.stdout: socat unzip xmlstarlet zip 2024-04-18T10:02:12.055 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T10:02:12.056 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T10:02:12.056 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T10:02:12.056 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T10:02:12.056 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-18T10:02:12.057 INFO:teuthology.orchestra.run.smithi062.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-18T10:02:12.057 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T10:02:12.057 INFO:teuthology.orchestra.run.smithi062.stdout:The following additional packages will be installed: 2024-04-18T10:02:12.057 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-18T10:02:12.058 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-18T10:02:12.058 INFO:teuthology.orchestra.run.smithi062.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-18T10:02:12.058 INFO:teuthology.orchestra.run.smithi062.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-18T10:02:12.059 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-18T10:02:12.059 INFO:teuthology.orchestra.run.smithi062.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-18T10:02:12.059 INFO:teuthology.orchestra.run.smithi062.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-18T10:02:12.059 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-18T10:02:12.059 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-18T10:02:12.059 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-18T10:02:12.059 INFO:teuthology.orchestra.run.smithi062.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-18T10:02:12.059 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-18T10:02:12.060 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-18T10:02:12.060 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-18T10:02:12.060 INFO:teuthology.orchestra.run.smithi062.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-18T10:02:12.060 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-18T10:02:12.060 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-18T10:02:12.060 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-18T10:02:12.060 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-18T10:02:12.060 INFO:teuthology.orchestra.run.smithi062.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-18T10:02:12.060 INFO:teuthology.orchestra.run.smithi062.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-18T10:02:12.060 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-18T10:02:12.060 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-18T10:02:12.060 INFO:teuthology.orchestra.run.smithi062.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-18T10:02:12.061 INFO:teuthology.orchestra.run.smithi062.stdout:Suggested packages: 2024-04-18T10:02:12.061 INFO:teuthology.orchestra.run.smithi062.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-18T10:02:12.061 INFO:teuthology.orchestra.run.smithi062.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-18T10:02:12.061 INFO:teuthology.orchestra.run.smithi062.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-18T10:02:12.061 INFO:teuthology.orchestra.run.smithi062.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-18T10:02:12.061 INFO:teuthology.orchestra.run.smithi062.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-18T10:02:12.061 INFO:teuthology.orchestra.run.smithi062.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-18T10:02:12.061 INFO:teuthology.orchestra.run.smithi062.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-18T10:02:12.061 INFO:teuthology.orchestra.run.smithi062.stdout:Recommended packages: 2024-04-18T10:02:12.062 INFO:teuthology.orchestra.run.smithi062.stdout: btrfs-tools 2024-04-18T10:02:12.100 INFO:teuthology.orchestra.run.smithi145.stdout:0 upgraded, 124 newly installed, 0 to remove and 302 not upgraded. 2024-04-18T10:02:12.100 INFO:teuthology.orchestra.run.smithi145.stdout:Need to get 182 MB of archives. 2024-04-18T10:02:12.100 INFO:teuthology.orchestra.run.smithi145.stdout:After this operation, 830 MB of additional disk space will be used. 2024-04-18T10:02:12.100 INFO:teuthology.orchestra.run.smithi145.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-18T10:02:12.174 INFO:teuthology.orchestra.run.smithi062.stdout:The following NEW packages will be installed: 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-18T10:02:12.175 INFO:teuthology.orchestra.run.smithi062.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-18T10:02:12.176 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-18T10:02:12.177 INFO:teuthology.orchestra.run.smithi062.stdout: socat unzip xmlstarlet zip 2024-04-18T10:02:12.182 INFO:teuthology.orchestra.run.smithi145.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-18T10:02:12.228 INFO:teuthology.orchestra.run.smithi145.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-18T10:02:12.238 INFO:teuthology.orchestra.run.smithi062.stdout:0 upgraded, 124 newly installed, 0 to remove and 302 not upgraded. 2024-04-18T10:02:12.239 INFO:teuthology.orchestra.run.smithi062.stdout:Need to get 182 MB of archives. 2024-04-18T10:02:12.239 INFO:teuthology.orchestra.run.smithi062.stdout:After this operation, 830 MB of additional disk space will be used. 2024-04-18T10:02:12.239 INFO:teuthology.orchestra.run.smithi062.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-18T10:02:12.259 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-18T10:02:12.269 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-18T10:02:12.316 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-18T10:02:12.340 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-18T10:02:12.343 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-18T10:02:12.356 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-18T10:02:12.411 INFO:teuthology.orchestra.run.smithi145.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3115-ga6549451-1jammy [3,556 kB] 2024-04-18T10:02:12.412 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-18T10:02:12.416 INFO:teuthology.orchestra.run.smithi062.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-18T10:02:12.429 INFO:teuthology.orchestra.run.smithi062.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-18T10:02:12.430 INFO:teuthology.orchestra.run.smithi145.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-18T10:02:12.433 INFO:teuthology.orchestra.run.smithi062.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-18T10:02:12.434 INFO:teuthology.orchestra.run.smithi062.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-18T10:02:12.442 INFO:teuthology.orchestra.run.smithi062.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-18T10:02:12.445 INFO:teuthology.orchestra.run.smithi062.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-18T10:02:12.447 INFO:teuthology.orchestra.run.smithi062.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-18T10:02:12.448 INFO:teuthology.orchestra.run.smithi062.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-18T10:02:12.448 INFO:teuthology.orchestra.run.smithi062.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-18T10:02:12.456 INFO:teuthology.orchestra.run.smithi062.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-18T10:02:12.458 INFO:teuthology.orchestra.run.smithi062.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-18T10:02:12.471 INFO:teuthology.orchestra.run.smithi062.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-18T10:02:12.472 INFO:teuthology.orchestra.run.smithi062.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-18T10:02:12.472 INFO:teuthology.orchestra.run.smithi062.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-18T10:02:12.474 INFO:teuthology.orchestra.run.smithi062.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-18T10:02:12.474 INFO:teuthology.orchestra.run.smithi062.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-18T10:02:12.474 INFO:teuthology.orchestra.run.smithi062.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-18T10:02:12.474 INFO:teuthology.orchestra.run.smithi062.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-18T10:02:12.475 INFO:teuthology.orchestra.run.smithi062.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-18T10:02:12.495 INFO:teuthology.orchestra.run.smithi062.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-18T10:02:12.495 INFO:teuthology.orchestra.run.smithi062.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-18T10:02:12.498 INFO:teuthology.orchestra.run.smithi062.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-18T10:02:12.498 INFO:teuthology.orchestra.run.smithi062.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-18T10:02:12.499 INFO:teuthology.orchestra.run.smithi062.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-18T10:02:12.499 INFO:teuthology.orchestra.run.smithi062.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-18T10:02:12.500 INFO:teuthology.orchestra.run.smithi062.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-18T10:02:12.500 INFO:teuthology.orchestra.run.smithi062.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-18T10:02:12.501 INFO:teuthology.orchestra.run.smithi062.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-18T10:02:12.517 INFO:teuthology.orchestra.run.smithi062.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-18T10:02:12.522 INFO:teuthology.orchestra.run.smithi062.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-18T10:02:12.523 INFO:teuthology.orchestra.run.smithi062.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-18T10:02:12.524 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-18T10:02:12.528 INFO:teuthology.orchestra.run.smithi062.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-18T10:02:12.528 INFO:teuthology.orchestra.run.smithi062.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-18T10:02:12.529 INFO:teuthology.orchestra.run.smithi062.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-18T10:02:12.530 INFO:teuthology.orchestra.run.smithi062.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-18T10:02:12.531 INFO:teuthology.orchestra.run.smithi062.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-18T10:02:12.533 INFO:teuthology.orchestra.run.smithi062.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-18T10:02:12.536 INFO:teuthology.orchestra.run.smithi062.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-18T10:02:12.538 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-18T10:02:12.554 INFO:teuthology.orchestra.run.smithi062.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-18T10:02:12.554 INFO:teuthology.orchestra.run.smithi062.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-18T10:02:12.557 INFO:teuthology.orchestra.run.smithi062.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-18T10:02:12.557 INFO:teuthology.orchestra.run.smithi062.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-18T10:02:12.558 INFO:teuthology.orchestra.run.smithi062.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-18T10:02:12.560 INFO:teuthology.orchestra.run.smithi062.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-18T10:02:12.590 INFO:teuthology.orchestra.run.smithi145.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-18T10:02:12.613 INFO:teuthology.orchestra.run.smithi062.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-18T10:02:12.614 INFO:teuthology.orchestra.run.smithi062.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-18T10:02:12.615 INFO:teuthology.orchestra.run.smithi062.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-18T10:02:12.616 INFO:teuthology.orchestra.run.smithi062.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-18T10:02:12.616 INFO:teuthology.orchestra.run.smithi062.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-18T10:02:12.634 INFO:teuthology.orchestra.run.smithi062.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-18T10:02:12.636 INFO:teuthology.orchestra.run.smithi062.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-18T10:02:12.636 INFO:teuthology.orchestra.run.smithi062.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-18T10:02:12.651 INFO:teuthology.orchestra.run.smithi062.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-18T10:02:12.651 INFO:teuthology.orchestra.run.smithi062.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-18T10:02:12.654 INFO:teuthology.orchestra.run.smithi062.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-18T10:02:12.655 INFO:teuthology.orchestra.run.smithi062.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-18T10:02:12.655 INFO:teuthology.orchestra.run.smithi062.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-18T10:02:12.656 INFO:teuthology.orchestra.run.smithi062.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-18T10:02:12.665 INFO:teuthology.orchestra.run.smithi062.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3115-ga6549451-1jammy [3,556 kB] 2024-04-18T10:02:12.674 INFO:teuthology.orchestra.run.smithi062.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-18T10:02:12.675 INFO:teuthology.orchestra.run.smithi062.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-18T10:02:12.678 INFO:teuthology.orchestra.run.smithi062.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-18T10:02:12.679 INFO:teuthology.orchestra.run.smithi062.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-18T10:02:12.681 INFO:teuthology.orchestra.run.smithi062.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-18T10:02:12.681 INFO:teuthology.orchestra.run.smithi062.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-18T10:02:12.693 INFO:teuthology.orchestra.run.smithi062.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-18T10:02:12.696 INFO:teuthology.orchestra.run.smithi062.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-18T10:02:12.696 INFO:teuthology.orchestra.run.smithi062.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-18T10:02:12.697 INFO:teuthology.orchestra.run.smithi062.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-18T10:02:12.713 INFO:teuthology.orchestra.run.smithi062.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-18T10:02:12.717 INFO:teuthology.orchestra.run.smithi062.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-18T10:02:12.718 INFO:teuthology.orchestra.run.smithi062.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-18T10:02:12.720 INFO:teuthology.orchestra.run.smithi062.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-18T10:02:12.721 INFO:teuthology.orchestra.run.smithi062.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-18T10:02:12.721 INFO:teuthology.orchestra.run.smithi062.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-18T10:02:12.732 INFO:teuthology.orchestra.run.smithi062.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-18T10:02:12.733 INFO:teuthology.orchestra.run.smithi062.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-18T10:02:12.734 INFO:teuthology.orchestra.run.smithi062.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-18T10:02:12.740 INFO:teuthology.orchestra.run.smithi062.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-18T10:02:12.751 INFO:teuthology.orchestra.run.smithi062.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-18T10:02:12.752 INFO:teuthology.orchestra.run.smithi062.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-18T10:02:12.754 INFO:teuthology.orchestra.run.smithi062.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-18T10:02:12.754 INFO:teuthology.orchestra.run.smithi062.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-18T10:02:12.763 INFO:teuthology.orchestra.run.smithi145.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-18T10:02:12.772 INFO:teuthology.orchestra.run.smithi145.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-18T10:02:12.800 INFO:teuthology.orchestra.run.smithi145.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-18T10:02:12.808 INFO:teuthology.orchestra.run.smithi145.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-18T10:02:12.972 INFO:teuthology.orchestra.run.smithi145.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-18T10:02:13.543 INFO:teuthology.orchestra.run.smithi145.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-18T10:02:13.558 INFO:teuthology.orchestra.run.smithi145.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-18T10:02:13.573 INFO:teuthology.orchestra.run.smithi145.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-18T10:02:13.582 INFO:teuthology.orchestra.run.smithi145.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-18T10:02:13.582 INFO:teuthology.orchestra.run.smithi145.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-18T10:02:13.599 INFO:teuthology.orchestra.run.smithi145.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-18T10:02:13.618 INFO:teuthology.orchestra.run.smithi145.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3115-ga6549451-1jammy [3,224 kB] 2024-04-18T10:02:13.637 INFO:teuthology.orchestra.run.smithi145.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-18T10:02:13.650 INFO:teuthology.orchestra.run.smithi145.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-18T10:02:13.654 INFO:teuthology.orchestra.run.smithi145.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-18T10:02:13.654 INFO:teuthology.orchestra.run.smithi145.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-18T10:02:13.748 INFO:teuthology.orchestra.run.smithi062.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3115-ga6549451-1jammy [3,224 kB] 2024-04-18T10:02:13.824 INFO:teuthology.orchestra.run.smithi145.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-18T10:02:13.907 INFO:teuthology.orchestra.run.smithi145.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-18T10:02:13.931 INFO:teuthology.orchestra.run.smithi145.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-18T10:02:13.988 INFO:teuthology.orchestra.run.smithi145.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-18T10:02:14.031 INFO:teuthology.orchestra.run.smithi145.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-18T10:02:14.054 INFO:teuthology.orchestra.run.smithi145.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-18T10:02:14.082 INFO:teuthology.orchestra.run.smithi145.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-18T10:02:14.263 INFO:teuthology.orchestra.run.smithi062.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3115-ga6549451-1jammy [712 kB] 2024-04-18T10:02:14.263 INFO:teuthology.orchestra.run.smithi145.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-18T10:02:14.279 INFO:teuthology.orchestra.run.smithi145.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-18T10:02:14.286 INFO:teuthology.orchestra.run.smithi145.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-18T10:02:14.291 INFO:teuthology.orchestra.run.smithi145.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-18T10:02:14.362 INFO:teuthology.orchestra.run.smithi062.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3115-ga6549451-1jammy [357 kB] 2024-04-18T10:02:14.395 INFO:teuthology.orchestra.run.smithi062.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3115-ga6549451-1jammy [32.8 kB] 2024-04-18T10:02:14.433 INFO:teuthology.orchestra.run.smithi062.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3115-ga6549451-1jammy [180 kB] 2024-04-18T10:02:14.446 INFO:teuthology.orchestra.run.smithi062.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3115-ga6549451-1jammy [66.9 kB] 2024-04-18T10:02:14.451 INFO:teuthology.orchestra.run.smithi062.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3115-ga6549451-1jammy [330 kB] 2024-04-18T10:02:14.474 INFO:teuthology.orchestra.run.smithi062.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3115-ga6549451-1jammy [6,890 kB] 2024-04-18T10:02:14.476 INFO:teuthology.orchestra.run.smithi145.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-18T10:02:14.498 INFO:teuthology.orchestra.run.smithi145.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3115-ga6549451-1jammy [712 kB] 2024-04-18T10:02:14.560 INFO:teuthology.orchestra.run.smithi145.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-18T10:02:14.574 INFO:teuthology.orchestra.run.smithi145.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-18T10:02:14.665 INFO:teuthology.orchestra.run.smithi145.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3115-ga6549451-1jammy [357 kB] 2024-04-18T10:02:14.749 INFO:teuthology.orchestra.run.smithi145.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3115-ga6549451-1jammy [32.8 kB] 2024-04-18T10:02:14.751 INFO:teuthology.orchestra.run.smithi145.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-18T10:02:14.752 INFO:teuthology.orchestra.run.smithi145.stdout:Get:43 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3115-ga6549451-1jammy [180 kB] 2024-04-18T10:02:14.798 INFO:teuthology.orchestra.run.smithi145.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-18T10:02:14.811 INFO:teuthology.orchestra.run.smithi145.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-18T10:02:14.822 INFO:teuthology.orchestra.run.smithi145.stdout:Get:46 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3115-ga6549451-1jammy [66.9 kB] 2024-04-18T10:02:14.829 INFO:teuthology.orchestra.run.smithi145.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3115-ga6549451-1jammy [330 kB] 2024-04-18T10:02:14.905 INFO:teuthology.orchestra.run.smithi145.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3115-ga6549451-1jammy [6,890 kB] 2024-04-18T10:02:14.972 INFO:teuthology.orchestra.run.smithi145.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-18T10:02:14.973 INFO:teuthology.orchestra.run.smithi145.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-18T10:02:14.978 INFO:teuthology.orchestra.run.smithi145.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-18T10:02:14.983 INFO:teuthology.orchestra.run.smithi145.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-18T10:02:14.997 INFO:teuthology.orchestra.run.smithi145.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-18T10:02:15.182 INFO:teuthology.orchestra.run.smithi145.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-18T10:02:15.188 INFO:teuthology.orchestra.run.smithi062.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3115-ga6549451-1jammy [112 kB] 2024-04-18T10:02:15.196 INFO:teuthology.orchestra.run.smithi062.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3115-ga6549451-1jammy [466 kB] 2024-04-18T10:02:15.225 INFO:teuthology.orchestra.run.smithi062.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3115-ga6549451-1jammy [25.8 MB] 2024-04-18T10:02:15.446 INFO:teuthology.orchestra.run.smithi145.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-18T10:02:15.561 INFO:teuthology.orchestra.run.smithi145.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-18T10:02:15.604 INFO:teuthology.orchestra.run.smithi145.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-18T10:02:15.699 INFO:teuthology.orchestra.run.smithi145.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-18T10:02:15.710 INFO:teuthology.orchestra.run.smithi145.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-18T10:02:15.720 INFO:teuthology.orchestra.run.smithi145.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-18T10:02:15.759 INFO:teuthology.orchestra.run.smithi145.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-18T10:02:16.030 INFO:teuthology.orchestra.run.smithi145.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-18T10:02:16.032 INFO:teuthology.orchestra.run.smithi145.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-18T10:02:16.034 INFO:teuthology.orchestra.run.smithi145.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-18T10:02:16.199 INFO:teuthology.orchestra.run.smithi145.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-18T10:02:16.523 INFO:teuthology.orchestra.run.smithi145.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3115-ga6549451-1jammy [112 kB] 2024-04-18T10:02:16.525 INFO:teuthology.orchestra.run.smithi145.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-18T10:02:16.526 INFO:teuthology.orchestra.run.smithi145.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3115-ga6549451-1jammy [466 kB] 2024-04-18T10:02:16.600 INFO:teuthology.orchestra.run.smithi145.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3115-ga6549451-1jammy [25.8 MB] 2024-04-18T10:02:16.932 INFO:teuthology.orchestra.run.smithi145.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-18T10:02:16.941 INFO:teuthology.orchestra.run.smithi145.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-18T10:02:16.942 INFO:teuthology.orchestra.run.smithi145.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-18T10:02:17.012 INFO:teuthology.orchestra.run.smithi145.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-18T10:02:17.012 INFO:teuthology.orchestra.run.smithi145.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-18T10:02:17.014 INFO:teuthology.orchestra.run.smithi145.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-18T10:02:17.016 INFO:teuthology.orchestra.run.smithi145.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-18T10:02:17.017 INFO:teuthology.orchestra.run.smithi145.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-18T10:02:17.183 INFO:teuthology.orchestra.run.smithi145.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-18T10:02:17.240 INFO:teuthology.orchestra.run.smithi062.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3115-ga6549451-1jammy [5,092 kB] 2024-04-18T10:02:17.278 INFO:teuthology.orchestra.run.smithi145.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-18T10:02:17.377 INFO:teuthology.orchestra.run.smithi145.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-18T10:02:17.466 INFO:teuthology.orchestra.run.smithi062.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3115-ga6549451-1jammy [244 kB] 2024-04-18T10:02:17.476 INFO:teuthology.orchestra.run.smithi062.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3115-ga6549451-1jammy [124 kB] 2024-04-18T10:02:17.481 INFO:teuthology.orchestra.run.smithi062.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3115-ga6549451-1jammy [1,527 kB] 2024-04-18T10:02:17.560 INFO:teuthology.orchestra.run.smithi062.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3115-ga6549451-1jammy [6,258 kB] 2024-04-18T10:02:17.625 INFO:teuthology.orchestra.run.smithi145.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-18T10:02:17.668 INFO:teuthology.orchestra.run.smithi145.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-18T10:02:17.692 INFO:teuthology.orchestra.run.smithi145.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-18T10:02:17.700 INFO:teuthology.orchestra.run.smithi145.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-18T10:02:17.745 INFO:teuthology.orchestra.run.smithi145.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-18T10:02:17.769 INFO:teuthology.orchestra.run.smithi145.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-18T10:02:17.771 INFO:teuthology.orchestra.run.smithi145.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-18T10:02:17.778 INFO:teuthology.orchestra.run.smithi145.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-18T10:02:17.911 INFO:teuthology.orchestra.run.smithi062.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3115-ga6549451-1jammy [22.8 MB] 2024-04-18T10:02:17.945 INFO:teuthology.orchestra.run.smithi145.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-18T10:02:18.486 INFO:teuthology.orchestra.run.smithi145.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-18T10:02:18.491 INFO:teuthology.orchestra.run.smithi145.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-18T10:02:18.531 INFO:teuthology.orchestra.run.smithi145.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-18T10:02:18.531 INFO:teuthology.orchestra.run.smithi145.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-18T10:02:18.537 INFO:teuthology.orchestra.run.smithi145.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-18T10:02:18.540 INFO:teuthology.orchestra.run.smithi145.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-18T10:02:18.583 INFO:teuthology.orchestra.run.smithi145.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-18T10:02:18.590 INFO:teuthology.orchestra.run.smithi145.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-18T10:02:18.662 INFO:teuthology.orchestra.run.smithi145.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-18T10:02:18.663 INFO:teuthology.orchestra.run.smithi145.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-18T10:02:18.828 INFO:teuthology.orchestra.run.smithi145.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-18T10:02:19.078 INFO:teuthology.orchestra.run.smithi062.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3115-ga6549451-1jammy [14.0 kB] 2024-04-18T10:02:19.079 INFO:teuthology.orchestra.run.smithi062.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3115-ga6549451-1jammy [870 kB] 2024-04-18T10:02:19.095 INFO:teuthology.orchestra.run.smithi062.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3115-ga6549451-1jammy [2,436 kB] 2024-04-18T10:02:19.137 INFO:teuthology.orchestra.run.smithi062.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3115-ga6549451-1jammy [791 kB] 2024-04-18T10:02:19.177 INFO:teuthology.orchestra.run.smithi062.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3115-ga6549451-1jammy [148 kB] 2024-04-18T10:02:19.182 INFO:teuthology.orchestra.run.smithi062.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3115-ga6549451-1jammy [2,297 kB] 2024-04-18T10:02:19.263 INFO:teuthology.orchestra.run.smithi145.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-18T10:02:19.278 INFO:teuthology.orchestra.run.smithi145.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-18T10:02:19.285 INFO:teuthology.orchestra.run.smithi062.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3115-ga6549451-1jammy [8,625 kB] 2024-04-18T10:02:19.653 INFO:teuthology.orchestra.run.smithi062.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3115-ga6549451-1jammy [14.2 kB] 2024-04-18T10:02:19.653 INFO:teuthology.orchestra.run.smithi062.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3115-ga6549451-1jammy [51.6 MB] 2024-04-18T10:02:20.369 INFO:teuthology.orchestra.run.smithi145.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3115-ga6549451-1jammy [5,092 kB] 2024-04-18T10:02:21.027 INFO:teuthology.orchestra.run.smithi145.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3115-ga6549451-1jammy [244 kB] 2024-04-18T10:02:21.037 INFO:teuthology.orchestra.run.smithi145.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3115-ga6549451-1jammy [124 kB] 2024-04-18T10:02:21.079 INFO:teuthology.orchestra.run.smithi145.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3115-ga6549451-1jammy [1,527 kB] 2024-04-18T10:02:21.260 INFO:teuthology.orchestra.run.smithi145.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3115-ga6549451-1jammy [6,258 kB] 2024-04-18T10:02:22.065 INFO:teuthology.orchestra.run.smithi145.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3115-ga6549451-1jammy [22.8 MB] 2024-04-18T10:02:22.357 INFO:teuthology.orchestra.run.smithi062.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3115-ga6549451-1jammy [124 kB] 2024-04-18T10:02:22.359 INFO:teuthology.orchestra.run.smithi062.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-3115-ga6549451-1jammy [232 kB] 2024-04-18T10:02:22.364 INFO:teuthology.orchestra.run.smithi062.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-3115-ga6549451-1jammy [34.0 kB] 2024-04-18T10:02:22.365 INFO:teuthology.orchestra.run.smithi062.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-3115-ga6549451-1jammy [23.4 kB] 2024-04-18T10:02:22.366 INFO:teuthology.orchestra.run.smithi062.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3115-ga6549451-1jammy [38.9 kB] 2024-04-18T10:02:22.367 INFO:teuthology.orchestra.run.smithi062.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3115-ga6549451-1jammy [13.6 MB] 2024-04-18T10:02:22.817 INFO:teuthology.orchestra.run.smithi062.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3115-ga6549451-1jammy [91.4 kB] 2024-04-18T10:02:23.471 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 182 MB in 11s (17.1 MB/s) 2024-04-18T10:02:23.615 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-18T10:02:23.711 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-18T10:02:23.717 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-18T10:02:23.763 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-18T10:02:24.033 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-18T10:02:24.060 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-18T10:02:24.091 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-18T10:02:24.219 INFO:teuthology.orchestra.run.smithi145.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3115-ga6549451-1jammy [14.0 kB] 2024-04-18T10:02:24.219 INFO:teuthology.orchestra.run.smithi145.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3115-ga6549451-1jammy [870 kB] 2024-04-18T10:02:24.236 INFO:teuthology.orchestra.run.smithi145.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3115-ga6549451-1jammy [2,436 kB] 2024-04-18T10:02:24.411 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cmd2. 2024-04-18T10:02:24.436 INFO:teuthology.orchestra.run.smithi145.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3115-ga6549451-1jammy [791 kB] 2024-04-18T10:02:24.437 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-18T10:02:24.478 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-18T10:02:24.491 INFO:teuthology.orchestra.run.smithi145.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3115-ga6549451-1jammy [148 kB] 2024-04-18T10:02:24.510 INFO:teuthology.orchestra.run.smithi145.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3115-ga6549451-1jammy [2,297 kB] 2024-04-18T10:02:24.702 INFO:teuthology.orchestra.run.smithi145.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3115-ga6549451-1jammy [8,625 kB] 2024-04-18T10:02:24.814 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-18T10:02:24.841 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-18T10:02:24.922 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-18T10:02:25.276 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-18T10:02:25.303 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-18T10:02:25.343 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-18T10:02:25.396 INFO:teuthology.orchestra.run.smithi145.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3115-ga6549451-1jammy [14.2 kB] 2024-04-18T10:02:25.449 INFO:teuthology.orchestra.run.smithi145.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3115-ga6549451-1jammy [51.6 MB] 2024-04-18T10:02:25.687 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-18T10:02:25.714 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-18T10:02:25.754 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-18T10:02:26.166 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-18T10:02:26.193 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T10:02:26.241 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T10:02:26.661 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-18T10:02:26.687 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T10:02:26.727 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T10:02:27.233 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-18T10:02:27.261 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T10:02:27.472 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T10:02:27.925 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-18T10:02:27.952 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-18T10:02:28.001 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-18T10:02:28.287 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librados2. 2024-04-18T10:02:28.303 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../010-librados2_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:28.337 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librados2 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:28.773 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librbd1. 2024-04-18T10:02:28.800 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../011-librbd1_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:28.818 INFO:teuthology.orchestra.run.smithi145.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3115-ga6549451-1jammy [124 kB] 2024-04-18T10:02:28.821 INFO:teuthology.orchestra.run.smithi145.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-3115-ga6549451-1jammy [232 kB] 2024-04-18T10:02:28.826 INFO:teuthology.orchestra.run.smithi145.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-3115-ga6549451-1jammy [34.0 kB] 2024-04-18T10:02:28.827 INFO:teuthology.orchestra.run.smithi145.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-3115-ga6549451-1jammy [23.4 kB] 2024-04-18T10:02:28.827 INFO:teuthology.orchestra.run.smithi145.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3115-ga6549451-1jammy [38.9 kB] 2024-04-18T10:02:28.839 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librbd1 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:28.866 INFO:teuthology.orchestra.run.smithi145.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3115-ga6549451-1jammy [13.6 MB] 2024-04-18T10:02:29.284 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libcephfs2. 2024-04-18T10:02:29.311 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:29.314 INFO:teuthology.orchestra.run.smithi145.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/main/a6549451253d26216d353246c0fe6f6d80f3b9cf/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3115-ga6549451-1jammy [91.4 kB] 2024-04-18T10:02:29.359 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libcephfs2 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:29.604 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rados. 2024-04-18T10:02:30.133 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 182 MB in 17s (10.4 MB/s) 2024-04-18T10:02:30.134 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../013-python3-rados_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:30.134 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rados (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:30.134 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-18T10:02:30.134 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:02:30.134 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-ceph-argparse (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:30.140 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-18T10:02:30.237 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-18T10:02:30.242 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-18T10:02:30.288 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-18T10:02:30.300 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cephfs. 2024-04-18T10:02:30.316 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:30.358 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cephfs (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:30.574 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-18T10:02:30.590 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-18T10:02:30.624 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-18T10:02:30.678 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-18T10:02:30.705 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:02:30.744 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-ceph-common (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:30.935 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cmd2. 2024-04-18T10:02:30.962 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-18T10:02:31.002 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-18T10:02:31.039 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-prettytable. 2024-04-18T10:02:31.066 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-18T10:02:31.097 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-18T10:02:31.347 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-18T10:02:31.373 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-18T10:02:31.375 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rbd. 2024-04-18T10:02:31.402 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:31.441 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rbd (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:31.455 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-18T10:02:31.775 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-18T10:02:31.778 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-18T10:02:31.802 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-18T10:02:31.805 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-18T10:02:31.842 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-18T10:02:31.844 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-18T10:02:32.122 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-18T10:02:32.138 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-18T10:02:32.145 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-18T10:02:32.171 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-18T10:02:32.180 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-18T10:02:32.211 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-18T10:02:32.492 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-18T10:02:32.519 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-18T10:02:32.550 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-18T10:02:32.632 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-18T10:02:32.658 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T10:02:32.706 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T10:02:32.889 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package lua5.1. 2024-04-18T10:02:32.916 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-18T10:02:32.955 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-18T10:02:33.110 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-18T10:02:33.136 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T10:02:33.176 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T10:02:33.267 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package lua-any. 2024-04-18T10:02:33.294 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-18T10:02:33.333 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-18T10:02:33.496 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-18T10:02:33.523 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-18T10:02:33.544 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package zip. 2024-04-18T10:02:33.554 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T10:02:33.568 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-18T10:02:33.610 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking zip (3.0-12build2) ... 2024-04-18T10:02:33.916 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-18T10:02:33.930 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package unzip. 2024-04-18T10:02:33.942 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-18T10:02:33.957 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-18T10:02:33.982 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-18T10:02:33.997 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-18T10:02:34.392 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package luarocks. 2024-04-18T10:02:34.420 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-18T10:02:34.459 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-18T10:02:34.493 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package librados2. 2024-04-18T10:02:34.520 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../010-librados2_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:34.560 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librados2 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:34.833 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package librgw2. 2024-04-18T10:02:34.860 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../027-librgw2_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:34.899 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking librgw2 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:34.980 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package librbd1. 2024-04-18T10:02:34.995 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../011-librbd1_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:35.030 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librbd1 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:35.399 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libcephfs2. 2024-04-18T10:02:35.425 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:35.466 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libcephfs2 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:35.468 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rgw. 2024-04-18T10:02:35.484 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:35.527 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rgw (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:35.743 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-rados. 2024-04-18T10:02:35.770 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../013-python3-rados_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:35.809 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-rados (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:35.880 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-18T10:02:35.907 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-18T10:02:35.946 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-18T10:02:36.138 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-18T10:02:36.164 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:02:36.199 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libradosstriper1. 2024-04-18T10:02:36.204 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-ceph-argparse (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:36.215 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:36.249 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libradosstriper1 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:36.457 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cephfs. 2024-04-18T10:02:36.472 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:36.515 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cephfs (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:36.669 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-common. 2024-04-18T10:02:36.696 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../031-ceph-common_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:36.735 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-common (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:36.809 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-18T10:02:36.836 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:02:36.876 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-ceph-common (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:37.254 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-prettytable. 2024-04-18T10:02:37.281 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-18T10:02:37.321 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-18T10:02:37.557 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-rbd. 2024-04-18T10:02:37.572 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:37.671 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-rbd (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:38.010 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-18T10:02:38.036 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-18T10:02:38.068 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-18T10:02:38.141 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-base. 2024-04-18T10:02:38.168 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../032-ceph-base_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:38.217 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-base (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:38.312 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-18T10:02:38.327 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-18T10:02:38.370 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-18T10:02:38.648 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-18T10:02:38.671 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-18T10:02:38.706 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-18T10:02:38.811 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-18T10:02:38.838 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-18T10:02:38.878 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-18T10:02:39.018 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package lua5.1. 2024-04-18T10:02:39.044 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-18T10:02:39.085 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-18T10:02:39.181 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cheroot. 2024-04-18T10:02:39.208 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-18T10:02:39.247 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-18T10:02:39.414 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package lua-any. 2024-04-18T10:02:39.442 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-18T10:02:39.473 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-18T10:02:39.575 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-18T10:02:39.602 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-18T10:02:39.642 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-18T10:02:39.709 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package zip. 2024-04-18T10:02:39.735 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-18T10:02:39.775 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking zip (3.0-12build2) ... 2024-04-18T10:02:39.953 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-18T10:02:39.980 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-18T10:02:40.019 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-18T10:02:40.104 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package unzip. 2024-04-18T10:02:40.130 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-18T10:02:40.170 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-18T10:02:40.255 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-18T10:02:40.282 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-18T10:02:40.313 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-18T10:02:40.541 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package luarocks. 2024-04-18T10:02:40.567 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-18T10:02:40.600 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-tempora. 2024-04-18T10:02:40.607 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-18T10:02:40.627 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-18T10:02:40.666 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-18T10:02:40.932 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package librgw2. 2024-04-18T10:02:40.959 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../027-librgw2_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:40.969 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-portend. 2024-04-18T10:02:40.990 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librgw2 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:40.996 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-18T10:02:41.035 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-18T10:02:41.289 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-18T10:02:41.315 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-18T10:02:41.355 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-18T10:02:41.543 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-rgw. 2024-04-18T10:02:41.569 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:41.609 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-rgw (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:41.641 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-18T10:02:41.668 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-18T10:02:41.707 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-18T10:02:41.946 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-18T10:02:41.972 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-18T10:02:42.012 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-18T10:02:42.273 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libradosstriper1. 2024-04-18T10:02:42.299 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:42.332 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libradosstriper1 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:42.820 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-common. 2024-04-18T10:02:42.846 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../031-ceph-common_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:42.893 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-common (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:42.936 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-natsort. 2024-04-18T10:02:42.964 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-18T10:02:42.994 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-18T10:02:43.289 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-logutils. 2024-04-18T10:02:43.316 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-18T10:02:43.355 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-18T10:02:43.642 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-18T10:02:43.669 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-18T10:02:43.708 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-18T10:02:43.977 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-mako. 2024-04-18T10:02:43.993 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-18T10:02:44.036 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-18T10:02:44.207 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-base. 2024-04-18T10:02:44.233 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../032-ceph-base_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:44.282 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-base (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:44.364 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-18T10:02:44.391 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-18T10:02:44.430 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-18T10:02:44.733 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-18T10:02:44.762 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-18T10:02:44.800 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-18T10:02:45.102 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-webob. 2024-04-18T10:02:45.130 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-18T10:02:45.170 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-18T10:02:45.539 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-waitress. 2024-04-18T10:02:45.566 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-18T10:02:45.608 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-18T10:02:45.869 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-18T10:02:45.896 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-18T10:02:45.916 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-tempita. 2024-04-18T10:02:45.936 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-18T10:02:45.944 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-18T10:02:45.983 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-18T10:02:46.238 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cheroot. 2024-04-18T10:02:46.244 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-paste. 2024-04-18T10:02:46.265 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-18T10:02:46.271 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-18T10:02:46.305 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-18T10:02:46.310 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-18T10:02:46.625 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-18T10:02:46.630 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-18T10:02:46.646 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-18T10:02:46.651 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-18T10:02:46.688 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-18T10:02:46.691 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-18T10:02:46.975 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-18T10:02:46.977 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-18T10:02:47.002 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-18T10:02:47.004 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-18T10:02:47.036 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-18T10:02:47.041 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-18T10:02:47.311 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-webtest. 2024-04-18T10:02:47.338 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-18T10:02:47.347 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-18T10:02:47.369 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-18T10:02:47.373 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-18T10:02:47.414 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-18T10:02:47.700 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-tempora. 2024-04-18T10:02:47.722 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pecan. 2024-04-18T10:02:47.726 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-18T10:02:47.749 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-18T10:02:47.766 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-18T10:02:47.788 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-18T10:02:48.069 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-portend. 2024-04-18T10:02:48.095 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-18T10:02:48.136 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-18T10:02:48.168 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-certifi. 2024-04-18T10:02:48.195 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-18T10:02:48.234 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-18T10:02:48.388 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-18T10:02:48.415 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-18T10:02:48.455 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-18T10:02:48.554 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-idna. 2024-04-18T10:02:48.581 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-18T10:02:48.620 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-18T10:02:48.767 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-18T10:02:48.793 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-18T10:02:48.833 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-18T10:02:48.932 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-urllib3. 2024-04-18T10:02:48.959 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-18T10:02:48.998 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-18T10:02:49.170 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-natsort. 2024-04-18T10:02:49.185 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-18T10:02:49.220 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-18T10:02:49.360 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-requests. 2024-04-18T10:02:49.387 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-18T10:02:49.426 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-18T10:02:49.490 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-logutils. 2024-04-18T10:02:49.516 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-18T10:02:49.548 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-18T10:02:49.704 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-18T10:02:49.731 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-18T10:02:49.762 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-18T10:02:49.817 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-18T10:02:49.844 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-18T10:02:49.894 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-18T10:02:50.065 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-18T10:02:50.092 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:02:50.140 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:50.196 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-mako. 2024-04-18T10:02:50.222 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-18T10:02:50.262 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-18T10:02:50.520 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-18T10:02:50.547 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:50.586 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:50.624 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-18T10:02:50.650 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-18T10:02:50.690 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-18T10:02:50.889 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-18T10:02:50.916 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-18T10:02:50.955 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-18T10:02:50.968 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-18T10:02:50.994 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-18T10:02:51.034 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-18T10:02:51.216 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-openssl. 2024-04-18T10:02:51.232 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-18T10:02:51.266 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-18T10:02:51.279 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-webob. 2024-04-18T10:02:51.305 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-18T10:02:51.345 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-18T10:02:51.603 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr. 2024-04-18T10:02:51.630 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:51.669 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:51.682 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-waitress. 2024-04-18T10:02:51.708 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-18T10:02:51.752 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-18T10:02:52.060 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-tempita. 2024-04-18T10:02:52.072 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mon. 2024-04-18T10:02:52.086 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-18T10:02:52.100 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:52.126 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-18T10:02:52.139 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mon (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:52.429 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-paste. 2024-04-18T10:02:52.455 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-18T10:02:52.495 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-18T10:02:52.717 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-osd. 2024-04-18T10:02:52.744 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:52.783 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-osd (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:52.832 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-18T10:02:52.847 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-18T10:02:52.890 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-18T10:02:53.193 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-18T10:02:53.219 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-18T10:02:53.259 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-18T10:02:53.521 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-webtest. 2024-04-18T10:02:53.550 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-18T10:02:53.579 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-18T10:02:53.845 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph. 2024-04-18T10:02:53.861 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../068-ceph_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:53.899 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pecan. 2024-04-18T10:02:53.903 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:53.926 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-18T10:02:53.965 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-18T10:02:54.214 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-fuse. 2024-04-18T10:02:54.241 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:54.280 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-fuse (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:54.303 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-certifi. 2024-04-18T10:02:54.330 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-18T10:02:54.370 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-18T10:02:54.667 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mds. 2024-04-18T10:02:54.681 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-idna. 2024-04-18T10:02:54.694 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:54.708 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-18T10:02:54.733 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mds (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:54.747 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-18T10:02:55.068 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-urllib3. 2024-04-18T10:02:55.094 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-18T10:02:55.094 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package cephadm. 2024-04-18T10:02:55.122 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../071-cephadm_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:55.134 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-18T10:02:55.152 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking cephadm (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:55.479 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-requests. 2024-04-18T10:02:55.497 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-18T10:02:55.505 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-18T10:02:55.525 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-18T10:02:55.545 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-18T10:02:55.564 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-18T10:02:55.823 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-18T10:02:55.850 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-18T10:02:55.858 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-18T10:02:55.874 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-18T10:02:55.890 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-18T10:02:55.908 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-18T10:02:56.168 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-18T10:02:56.183 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:02:56.226 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:56.598 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-18T10:02:56.624 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:56.664 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:56.954 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-babel. 2024-04-18T10:02:56.975 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-18T10:02:56.982 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-18T10:02:57.002 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-18T10:02:57.012 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-18T10:02:57.042 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-18T10:02:57.307 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-jinja2. 2024-04-18T10:02:57.334 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-18T10:02:57.336 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-openssl. 2024-04-18T10:02:57.351 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-18T10:02:57.365 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-18T10:02:57.386 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-18T10:02:57.673 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr. 2024-04-18T10:02:57.699 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:57.735 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-18T10:02:57.739 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:57.762 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:02:57.801 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:58.121 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-18T10:02:58.135 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mon. 2024-04-18T10:02:58.149 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-18T10:02:58.161 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:58.179 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-18T10:02:58.201 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mon (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:58.449 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-routes. 2024-04-18T10:02:58.476 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-18T10:02:58.515 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-18T10:02:58.738 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-osd. 2024-04-18T10:02:58.764 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:58.793 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-18T10:02:58.796 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-osd (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:58.821 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:02:58.851 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:02:59.849 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph. 2024-04-18T10:02:59.865 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../068-ceph_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:02:59.907 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:00.165 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-18T10:03:00.193 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-18T10:03:00.231 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-18T10:03:00.418 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-fuse. 2024-04-18T10:03:00.434 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:00.476 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-fuse (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:00.727 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-joblib. 2024-04-18T10:03:00.755 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-18T10:03:00.793 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-18T10:03:00.838 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mds. 2024-04-18T10:03:00.864 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:00.929 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mds (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:01.096 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-18T10:03:01.124 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-18T10:03:01.248 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-18T10:03:01.282 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package cephadm. 2024-04-18T10:03:01.297 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../071-cephadm_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:01.332 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking cephadm (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:01.560 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-sklearn. 2024-04-18T10:03:01.587 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-18T10:03:01.626 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-18T10:03:01.643 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-18T10:03:01.670 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-18T10:03:01.710 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-18T10:03:01.987 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-18T10:03:02.003 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-18T10:03:02.045 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-18T10:03:02.235 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-18T10:03:02.252 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:03:02.284 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:03.025 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-babel. 2024-04-18T10:03:03.042 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-18T10:03:03.076 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-18T10:03:03.346 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-cachetools. 2024-04-18T10:03:03.374 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-18T10:03:03.429 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-18T10:03:03.537 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-jinja2. 2024-04-18T10:03:03.564 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-18T10:03:03.603 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-18T10:03:03.732 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-18T10:03:03.760 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-18T10:03:03.799 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-18T10:03:03.948 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-18T10:03:03.975 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:03:04.014 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:04.144 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-18T10:03:04.171 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-18T10:03:04.201 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-18T10:03:04.310 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-18T10:03:04.336 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-18T10:03:04.376 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-18T10:03:04.488 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-rsa. 2024-04-18T10:03:04.516 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-18T10:03:04.554 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-18T10:03:04.679 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-routes. 2024-04-18T10:03:04.706 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-18T10:03:04.745 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-18T10:03:04.825 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-google-auth. 2024-04-18T10:03:04.852 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-18T10:03:04.883 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-18T10:03:05.074 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-18T10:03:05.100 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:03:05.148 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:05.178 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-18T10:03:05.206 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-18T10:03:05.244 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-18T10:03:05.489 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-websocket. 2024-04-18T10:03:05.517 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-18T10:03:05.564 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-18T10:03:05.884 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-18T10:03:05.912 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-18T10:03:05.950 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-18T10:03:06.429 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-18T10:03:06.456 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-18T10:03:06.495 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-18T10:03:06.547 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-18T10:03:06.563 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:03:06.605 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:06.925 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-18T10:03:06.952 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-18T10:03:06.998 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-joblib. 2024-04-18T10:03:07.007 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-18T10:03:07.026 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-18T10:03:07.082 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-18T10:03:07.286 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-18T10:03:07.314 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-18T10:03:07.344 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-18T10:03:07.452 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-18T10:03:07.479 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-18T10:03:07.604 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-18T10:03:07.613 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package jq. 2024-04-18T10:03:07.629 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-18T10:03:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-18T10:03:07.890 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-sklearn. 2024-04-18T10:03:07.917 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-18T10:03:07.949 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package socat. 2024-04-18T10:03:07.965 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-18T10:03:07.965 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-18T10:03:08.007 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-18T10:03:08.318 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package xmlstarlet. 2024-04-18T10:03:08.335 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-18T10:03:08.376 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-18T10:03:08.574 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-18T10:03:08.602 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:03:08.640 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:08.722 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-test. 2024-04-18T10:03:08.739 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../099-ceph-test_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:08.780 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-test (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:09.585 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cachetools. 2024-04-18T10:03:09.612 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-18T10:03:09.651 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-18T10:03:09.921 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-18T10:03:09.948 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-18T10:03:09.979 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-18T10:03:10.275 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-18T10:03:10.302 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-18T10:03:10.341 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-18T10:03:10.836 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-rsa. 2024-04-18T10:03:10.863 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-18T10:03:10.902 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-18T10:03:11.034 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ceph-volume. 2024-04-18T10:03:11.051 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:03:11.092 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ceph-volume (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:11.223 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-google-auth. 2024-04-18T10:03:11.249 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-18T10:03:11.289 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-18T10:03:11.422 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-18T10:03:11.450 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:11.488 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking cephfs-mirror (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:11.609 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-18T10:03:11.636 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-18T10:03:11.675 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-18T10:03:11.785 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-colorama. 2024-04-18T10:03:11.813 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-18T10:03:11.851 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-18T10:03:11.979 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-websocket. 2024-04-18T10:03:12.006 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-18T10:03:12.045 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-18T10:03:12.112 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package cephfs-shell. 2024-04-18T10:03:12.128 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:03:12.162 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking cephfs-shell (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:12.332 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-18T10:03:12.359 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-18T10:03:12.390 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-18T10:03:12.397 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package cephfs-top. 2024-04-18T10:03:12.414 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:03:12.447 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking cephfs-top (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:12.700 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-18T10:03:12.726 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:12.766 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libcephfs-dev (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:13.027 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-18T10:03:13.046 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-18T10:03:13.056 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-18T10:03:13.073 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:03:13.085 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-18T10:03:13.112 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:13.398 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-18T10:03:13.406 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-18T10:03:13.425 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-18T10:03:13.435 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-18T10:03:13.456 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-18T10:03:13.473 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-18T10:03:13.818 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-18T10:03:13.845 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-18T10:03:13.884 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-18T10:03:14.127 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package nvme-cli. 2024-04-18T10:03:14.145 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package jq. 2024-04-18T10:03:14.155 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-18T10:03:14.161 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-18T10:03:14.203 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-18T10:03:14.210 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-18T10:03:14.456 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package socat. 2024-04-18T10:03:14.479 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-18T10:03:14.523 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-18T10:03:14.648 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package pkg-config. 2024-04-18T10:03:14.677 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-18T10:03:14.714 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-18T10:03:14.792 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package xmlstarlet. 2024-04-18T10:03:14.808 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-18T10:03:14.842 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-18T10:03:14.959 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-18T10:03:14.976 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-18T10:03:15.009 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-18T10:03:15.172 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-test. 2024-04-18T10:03:15.199 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../099-ceph-test_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:15.238 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-test (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:15.387 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-18T10:03:15.416 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-18T10:03:15.454 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-18T10:03:15.756 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pastescript. 2024-04-18T10:03:15.785 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-18T10:03:15.823 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-18T10:03:16.143 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pluggy. 2024-04-18T10:03:16.171 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-18T10:03:16.209 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-18T10:03:16.520 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-psutil. 2024-04-18T10:03:16.549 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-18T10:03:16.587 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-18T10:03:16.940 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-py. 2024-04-18T10:03:16.968 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-18T10:03:17.006 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-18T10:03:17.318 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pygments. 2024-04-18T10:03:17.346 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-18T10:03:17.384 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-18T10:03:17.509 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-volume. 2024-04-18T10:03:17.536 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:03:17.575 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-volume (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:17.897 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-18T10:03:17.925 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-18T10:03:17.955 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-18T10:03:17.963 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-18T10:03:17.982 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:18.021 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking cephfs-mirror (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:18.274 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-toml. 2024-04-18T10:03:18.302 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-18T10:03:18.326 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-colorama. 2024-04-18T10:03:18.340 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-18T10:03:18.353 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-18T10:03:18.384 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-18T10:03:18.593 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-pytest. 2024-04-18T10:03:18.595 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package cephfs-shell. 2024-04-18T10:03:18.610 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-18T10:03:18.611 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:03:18.643 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-18T10:03:18.653 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking cephfs-shell (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:18.889 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package cephfs-top. 2024-04-18T10:03:18.916 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-3115-ga6549451-1jammy_all.deb ... 2024-04-18T10:03:18.955 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking cephfs-top (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:18.988 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package python3-simplejson. 2024-04-18T10:03:19.016 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-18T10:03:19.054 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-18T10:03:19.208 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-18T10:03:19.235 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:19.274 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libcephfs-dev (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:19.332 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-18T10:03:19.349 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-18T10:03:19.390 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-18T10:03:19.586 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-18T10:03:19.613 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-18T10:03:19.652 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-18T10:03:19.948 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-18T10:03:19.972 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-18T10:03:20.006 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-18T10:03:20.028 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package radosgw. 2024-04-18T10:03:20.056 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../122-radosgw_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:20.094 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking radosgw (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:20.419 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package nvme-cli. 2024-04-18T10:03:20.446 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-18T10:03:20.485 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-18T10:03:20.890 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package pkg-config. 2024-04-18T10:03:20.918 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-18T10:03:20.930 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package rbd-fuse. 2024-04-18T10:03:20.947 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:20.956 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-18T10:03:20.988 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking rbd-fuse (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:21.193 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-18T10:03:21.209 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-18T10:03:21.251 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-18T10:03:21.302 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-18T10:03:21.577 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-18T10:03:21.646 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-18T10:03:21.662 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-18T10:03:21.695 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-18T10:03:21.703 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-18T10:03:21.942 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-18T10:03:21.999 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pastescript. 2024-04-18T10:03:22.026 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-18T10:03:22.065 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-18T10:03:22.255 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-18T10:03:22.377 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pluggy. 2024-04-18T10:03:22.405 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-18T10:03:22.443 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-18T10:03:22.658 INFO:teuthology.orchestra.run.smithi062.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-18T10:03:22.721 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up cephadm (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:22.755 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-psutil. 2024-04-18T10:03:22.783 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-18T10:03:22.821 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-18T10:03:23.149 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-py. 2024-04-18T10:03:23.177 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-18T10:03:23.204 INFO:teuthology.orchestra.run.smithi062.stdout:Adding system user cephadm....done 2024-04-18T10:03:23.207 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-18T10:03:23.360 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-18T10:03:23.552 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pygments. 2024-04-18T10:03:23.568 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-18T10:03:23.610 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-18T10:03:23.745 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-18T10:03:23.996 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-18T10:03:24.106 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-18T10:03:24.122 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-18T10:03:24.134 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-18T10:03:24.172 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-18T10:03:24.483 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-toml. 2024-04-18T10:03:24.500 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-18T10:03:24.533 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-18T10:03:24.615 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-18T10:03:24.769 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pytest. 2024-04-18T10:03:24.786 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-18T10:03:24.827 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-18T10:03:24.874 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-18T10:03:24.992 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-18T10:03:25.164 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-simplejson. 2024-04-18T10:03:25.192 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-18T10:03:25.231 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-18T10:03:25.302 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-18T10:03:25.508 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-18T10:03:25.524 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-18T10:03:25.566 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-18T10:03:25.670 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-18T10:03:25.922 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-18T10:03:26.164 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-18T10:03:26.204 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package radosgw. 2024-04-18T10:03:26.220 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../122-radosgw_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:26.415 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking radosgw (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:26.441 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-18T10:03:26.692 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-18T10:03:26.960 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-18T10:03:27.266 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package rbd-fuse. 2024-04-18T10:03:27.293 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-3115-ga6549451-1jammy_amd64.deb ... 2024-04-18T10:03:27.311 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-18T10:03:27.332 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking rbd-fuse (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:27.612 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-18T10:03:27.670 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-ceph-argparse (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:27.862 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-18T10:03:27.938 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-18T10:03:27.988 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-18T10:03:28.064 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-18T10:03:28.190 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-18T10:03:28.263 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-18T10:03:28.449 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-18T10:03:28.592 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-18T10:03:28.595 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-18T10:03:28.851 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-18T10:03:29.047 INFO:teuthology.orchestra.run.smithi145.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-18T10:03:29.132 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up cephadm (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:29.215 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-18T10:03:29.248 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-18T10:03:29.321 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-18T10:03:29.447 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-18T10:03:29.831 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-18T10:03:29.949 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-18T10:03:29.973 INFO:teuthology.orchestra.run.smithi145.stdout:Adding system user cephadm....done 2024-04-18T10:03:30.147 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-18T10:03:30.217 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-18T10:03:30.449 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-18T10:03:30.460 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-18T10:03:30.700 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-18T10:03:30.760 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up zip (3.0-12build2) ... 2024-04-18T10:03:30.826 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-18T10:03:30.887 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-18T10:03:31.085 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-18T10:03:31.337 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-18T10:03:31.437 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-18T10:03:31.588 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-18T10:03:31.731 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-18T10:03:31.847 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-18T10:03:31.974 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-18T10:03:32.091 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-18T10:03:32.125 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-18T10:03:32.350 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-18T10:03:32.426 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-18T10:03:32.618 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-18T10:03:32.819 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-18T10:03:32.886 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-18T10:03:33.061 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-18T10:03:33.121 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-18T10:03:33.372 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-18T10:03:33.429 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-18T10:03:33.706 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-18T10:03:33.765 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-18T10:03:33.998 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-18T10:03:34.016 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-ceph-argparse (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:34.283 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-18T10:03:34.375 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-18T10:03:34.409 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-18T10:03:34.519 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-18T10:03:34.626 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-18T10:03:34.752 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-ceph-common (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:34.761 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-18T10:03:34.863 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-18T10:03:35.045 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-idna (3.3-1) ... 2024-04-18T10:03:35.097 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-18T10:03:35.362 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-18T10:03:35.473 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-18T10:03:35.504 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-18T10:03:35.567 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-18T10:03:35.693 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-18T10:03:36.069 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-18T10:03:36.179 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-18T10:03:36.447 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-18T10:03:36.635 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-18T10:03:36.706 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-18T10:03:36.870 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T10:03:36.979 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-18T10:03:37.040 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up zip (3.0-12build2) ... 2024-04-18T10:03:37.166 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-18T10:03:37.230 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-18T10:03:37.356 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-18T10:03:37.482 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-18T10:03:37.775 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-18T10:03:37.860 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-18T10:03:38.059 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-18T10:03:38.111 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-18T10:03:38.229 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-18T10:03:38.319 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T10:03:38.355 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-18T10:03:38.419 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-18T10:03:38.656 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-18T10:03:38.705 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-18T10:03:38.963 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-18T10:03:39.040 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-18T10:03:39.214 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-18T10:03:39.291 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-18T10:03:39.490 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-18T10:03:39.677 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-18T10:03:39.792 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-18T10:03:39.944 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-18T10:03:40.068 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-18T10:03:40.194 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-18T10:03:40.229 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-18T10:03:40.470 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-18T10:03:40.556 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-18T10:03:40.596 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-18T10:03:40.807 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-18T10:03:40.880 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-18T10:03:40.932 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-ceph-common (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:41.181 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-18T10:03:41.234 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-idna (3.3-1) ... 2024-04-18T10:03:41.432 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-18T10:03:41.543 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-18T10:03:41.558 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-18T10:03:41.684 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-18T10:03:41.773 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-18T10:03:42.011 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-18T10:03:42.137 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-18T10:03:42.547 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-18T10:03:42.724 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-18T10:03:42.798 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-18T10:03:42.957 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-18T10:03:42.976 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T10:03:43.110 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-18T10:03:43.216 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T10:03:43.343 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-18T10:03:43.361 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-18T10:03:43.487 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-18T10:03:43.613 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-18T10:03:43.669 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-18T10:03:43.931 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-18T10:03:43.961 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-18T10:03:44.174 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-18T10:03:44.237 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-18T10:03:44.440 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-18T10:03:44.442 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T10:03:44.568 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-18T10:03:44.716 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-18T10:03:44.869 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-18T10:03:44.993 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-18T10:03:45.112 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-18T10:03:45.261 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-18T10:03:45.371 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-18T10:03:45.622 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-18T10:03:45.637 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-18T10:03:45.857 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-18T10:03:45.939 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-18T10:03:46.116 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-18T10:03:46.251 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-18T10:03:46.291 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-18T10:03:46.442 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librados2 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:46.518 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-18T10:03:46.568 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:46.619 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-18T10:03:46.694 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-18T10:03:46.904 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-18T10:03:47.022 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libcephfs2 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:47.123 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libradosstriper1 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:47.224 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-18T10:03:47.230 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-18T10:03:47.481 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-18T10:03:47.576 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librbd1 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:47.599 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-18T10:03:47.702 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-modules-core (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:47.724 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-18T10:03:47.802 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-fuse (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:47.815 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-18T10:03:48.009 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-18T10:03:48.009 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-18T10:03:48.060 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-18T10:03:48.186 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-18T10:03:48.495 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libcephfs-dev (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:48.571 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-18T10:03:48.621 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rados (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:48.722 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up librgw2 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:48.823 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up cephfs-top (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:48.830 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-18T10:03:48.924 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-18T10:03:48.956 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-18T10:03:49.215 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-18T10:03:49.341 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-18T10:03:49.651 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-18T10:03:49.935 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-18T10:03:50.211 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-18T10:03:50.228 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rbd (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:50.354 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up rbd-fuse (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:50.448 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-18T10:03:50.479 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-rgw (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:50.597 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up python3-cephfs (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:50.716 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-18T10:03:50.723 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up cephfs-shell (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:50.849 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-common (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:50.976 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-18T10:03:51.227 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-18T10:03:51.376 INFO:teuthology.orchestra.run.smithi062.stdout:Adding group ceph....done 2024-04-18T10:03:51.595 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-18T10:03:51.718 INFO:teuthology.orchestra.run.smithi062.stdout:Adding system user ceph....done 2024-04-18T10:03:51.879 INFO:teuthology.orchestra.run.smithi062.stdout:Setting system user ceph properties....done 2024-04-18T10:03:51.888 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-18T10:03:51.892 INFO:teuthology.orchestra.run.smithi062.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-18T10:03:52.036 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-18T10:03:52.224 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-18T10:03:52.358 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librados2 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:52.406 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-18T10:03:52.459 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:52.561 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-18T10:03:52.850 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up cephfs-mirror (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:52.872 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libcephfs2 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:52.998 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libradosstriper1 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:53.066 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-18T10:03:53.067 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-18T10:03:53.124 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-18T10:03:53.400 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librbd1 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:53.501 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-test (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:53.502 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-modules-core (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:53.602 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-fuse (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:53.644 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up radosgw (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:53.832 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-18T10:03:53.832 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-18T10:03:54.082 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-18T10:03:54.083 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-18T10:03:54.304 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libcephfs-dev (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:54.421 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-rados (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:54.529 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-base (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:54.530 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librgw2 (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:54.640 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up cephfs-top (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:54.766 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-18T10:03:54.889 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-18T10:03:55.548 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mds (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:55.815 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-18T10:03:55.815 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-18T10:03:55.995 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-rbd (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:56.120 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up rbd-fuse (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:56.247 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-rgw (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:56.359 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:56.372 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cephfs (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:56.498 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up cephfs-shell (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:56.581 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-18T10:03:56.582 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-18T10:03:56.624 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-common (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:57.026 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-osd (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:57.158 INFO:teuthology.orchestra.run.smithi145.stdout:Adding group ceph....done 2024-04-18T10:03:57.290 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-18T10:03:57.291 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-18T10:03:57.535 INFO:teuthology.orchestra.run.smithi145.stdout:Adding system user ceph....done 2024-04-18T10:03:57.672 INFO:teuthology.orchestra.run.smithi145.stdout:Setting system user ceph properties....done 2024-04-18T10:03:57.688 INFO:teuthology.orchestra.run.smithi145.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-18T10:03:57.770 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:57.821 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-18T10:03:57.879 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:58.030 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mon (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:58.100 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-18T10:03:58.249 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-18T10:03:58.250 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-18T10:03:58.576 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up cephfs-mirror (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:58.706 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-cephadm (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:58.778 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-18T10:03:58.778 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-18T10:03:58.857 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:59.008 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-mgr-dashboard (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:59.184 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ceph-volume (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:59.268 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-test (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:59.395 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up radosgw (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:03:59.571 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-18T10:03:59.663 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-18T10:03:59.798 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-18T10:03:59.886 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-18T10:03:59.887 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-18T10:04:00.346 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-base (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:04:00.684 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-18T10:04:01.131 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mds (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:04:01.158 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-18T10:04:01.346 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-18T10:04:01.346 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-18T10:04:01.800 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:04:01.822 INFO:teuthology.orchestra.run.smithi062.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-18T10:04:01.826 DEBUG:teuthology.orchestra.run.smithi062:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-18T10:04:01.881 INFO:teuthology.orchestra.run.smithi062.stdout:Reading package lists... 2024-04-18T10:04:02.013 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-18T10:04:02.021 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-18T10:04:02.084 INFO:teuthology.orchestra.run.smithi062.stdout:Building dependency tree... 2024-04-18T10:04:02.085 INFO:teuthology.orchestra.run.smithi062.stdout:Reading state information... 2024-04-18T10:04:02.282 INFO:teuthology.orchestra.run.smithi062.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-18T10:04:02.283 INFO:teuthology.orchestra.run.smithi062.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-18T10:04:02.283 INFO:teuthology.orchestra.run.smithi062.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-18T10:04:02.283 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T10:04:02.283 INFO:teuthology.orchestra.run.smithi062.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T10:04:02.283 INFO:teuthology.orchestra.run.smithi062.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T10:04:02.283 INFO:teuthology.orchestra.run.smithi062.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T10:04:02.283 INFO:teuthology.orchestra.run.smithi062.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-18T10:04:02.284 INFO:teuthology.orchestra.run.smithi062.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-18T10:04:02.284 INFO:teuthology.orchestra.run.smithi062.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T10:04:02.284 INFO:teuthology.orchestra.run.smithi062.stdout:The following additional packages will be installed: 2024-04-18T10:04:02.284 INFO:teuthology.orchestra.run.smithi062.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-18T10:04:02.284 INFO:teuthology.orchestra.run.smithi062.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-18T10:04:02.285 INFO:teuthology.orchestra.run.smithi062.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-18T10:04:02.285 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-18T10:04:02.285 INFO:teuthology.orchestra.run.smithi062.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-18T10:04:02.286 INFO:teuthology.orchestra.run.smithi062.stdout:Suggested packages: 2024-04-18T10:04:02.286 INFO:teuthology.orchestra.run.smithi062.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-18T10:04:02.286 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-18T10:04:02.286 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-18T10:04:02.328 INFO:teuthology.orchestra.run.smithi062.stdout:The following NEW packages will be installed: 2024-04-18T10:04:02.328 INFO:teuthology.orchestra.run.smithi062.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-18T10:04:02.328 INFO:teuthology.orchestra.run.smithi062.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-18T10:04:02.329 INFO:teuthology.orchestra.run.smithi062.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-18T10:04:02.329 INFO:teuthology.orchestra.run.smithi062.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-18T10:04:02.329 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-18T10:04:02.329 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-18T10:04:02.329 INFO:teuthology.orchestra.run.smithi062.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-18T10:04:02.330 INFO:teuthology.orchestra.run.smithi062.stdout:The following packages will be upgraded: 2024-04-18T10:04:02.331 INFO:teuthology.orchestra.run.smithi062.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-18T10:04:02.412 INFO:teuthology.orchestra.run.smithi062.stdout:4 upgraded, 39 newly installed, 0 to remove and 298 not upgraded. 2024-04-18T10:04:02.412 INFO:teuthology.orchestra.run.smithi062.stdout:Need to get 55.7 MB of archives. 2024-04-18T10:04:02.413 INFO:teuthology.orchestra.run.smithi062.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-18T10:04:02.413 INFO:teuthology.orchestra.run.smithi062.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-18T10:04:02.568 INFO:teuthology.orchestra.run.smithi062.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-18T10:04:02.575 INFO:teuthology.orchestra.run.smithi062.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-18T10:04:02.615 INFO:teuthology.orchestra.run.smithi062.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-18T10:04:02.616 INFO:teuthology.orchestra.run.smithi062.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-18T10:04:02.616 INFO:teuthology.orchestra.run.smithi062.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-18T10:04:02.628 INFO:teuthology.orchestra.run.smithi062.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-18T10:04:02.629 INFO:teuthology.orchestra.run.smithi062.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-18T10:04:02.631 INFO:teuthology.orchestra.run.smithi062.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-18T10:04:02.632 INFO:teuthology.orchestra.run.smithi062.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-18T10:04:02.633 INFO:teuthology.orchestra.run.smithi062.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-18T10:04:02.634 INFO:teuthology.orchestra.run.smithi062.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-18T10:04:02.644 INFO:teuthology.orchestra.run.smithi062.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-18T10:04:02.644 INFO:teuthology.orchestra.run.smithi062.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-18T10:04:02.665 INFO:teuthology.orchestra.run.smithi062.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-18T10:04:02.667 INFO:teuthology.orchestra.run.smithi062.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-18T10:04:02.847 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-osd (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:04:03.081 INFO:teuthology.orchestra.run.smithi062.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-18T10:04:03.083 INFO:teuthology.orchestra.run.smithi062.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-18T10:04:03.126 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-18T10:04:03.127 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-18T10:04:03.137 INFO:teuthology.orchestra.run.smithi062.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-18T10:04:03.141 INFO:teuthology.orchestra.run.smithi062.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-18T10:04:03.141 INFO:teuthology.orchestra.run.smithi062.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-18T10:04:03.146 INFO:teuthology.orchestra.run.smithi062.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-18T10:04:03.148 INFO:teuthology.orchestra.run.smithi062.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-18T10:04:03.157 INFO:teuthology.orchestra.run.smithi062.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-18T10:04:03.157 INFO:teuthology.orchestra.run.smithi062.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-18T10:04:03.183 INFO:teuthology.orchestra.run.smithi062.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-18T10:04:03.183 INFO:teuthology.orchestra.run.smithi062.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-18T10:04:03.195 INFO:teuthology.orchestra.run.smithi062.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-18T10:04:03.216 INFO:teuthology.orchestra.run.smithi062.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-18T10:04:03.220 INFO:teuthology.orchestra.run.smithi062.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-18T10:04:03.220 INFO:teuthology.orchestra.run.smithi062.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-18T10:04:03.220 INFO:teuthology.orchestra.run.smithi062.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-18T10:04:03.221 INFO:teuthology.orchestra.run.smithi062.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-18T10:04:03.231 INFO:teuthology.orchestra.run.smithi062.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-18T10:04:03.233 INFO:teuthology.orchestra.run.smithi062.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-18T10:04:03.530 INFO:teuthology.orchestra.run.smithi062.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-18T10:04:03.531 INFO:teuthology.orchestra.run.smithi062.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-18T10:04:03.531 INFO:teuthology.orchestra.run.smithi062.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-18T10:04:03.531 INFO:teuthology.orchestra.run.smithi062.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-18T10:04:03.531 INFO:teuthology.orchestra.run.smithi062.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-18T10:04:03.531 INFO:teuthology.orchestra.run.smithi062.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-18T10:04:03.543 INFO:teuthology.orchestra.run.smithi062.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-18T10:04:03.543 INFO:teuthology.orchestra.run.smithi062.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-18T10:04:03.598 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:04:03.750 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:04:03.900 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mon (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:04:03.969 INFO:teuthology.orchestra.run.smithi062.stdout:Preconfiguring packages ... 2024-04-18T10:04:04.112 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-18T10:04:04.112 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-18T10:04:04.146 INFO:teuthology.orchestra.run.smithi062.stdout:Fetched 55.7 MB in 1s (46.3 MB/s) 2024-04-18T10:04:04.490 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173995 files and directories currently installed.) 2024-04-18T10:04:04.496 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-18T10:04:04.577 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-cephadm (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:04:04.640 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-18T10:04:04.719 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:04:04.845 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-dashboard (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:04:04.988 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-volume (19.0.0-3115-ga6549451-1jammy) ... 2024-04-18T10:04:05.255 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-18T10:04:05.340 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-18T10:04:05.442 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-18T10:04:05.586 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-18T10:04:06.250 INFO:teuthology.orchestra.run.smithi062.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-18T10:04:06.531 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package flex. 2024-04-18T10:04:06.637 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173995 files and directories currently installed.) 2024-04-18T10:04:06.643 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-18T10:04:06.670 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-18T10:04:06.687 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-18T10:04:07.163 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-18T10:04:07.335 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-18T10:04:07.338 INFO:teuthology.orchestra.run.smithi145.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-18T10:04:07.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-18T10:04:07.396 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-04-18T10:04:07.597 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2024-04-18T10:04:07.598 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2024-04-18T10:04:07.746 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-18T10:04:07.779 INFO:teuthology.orchestra.run.smithi145.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-18T10:04:07.779 INFO:teuthology.orchestra.run.smithi145.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-18T10:04:07.779 INFO:teuthology.orchestra.run.smithi145.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-18T10:04:07.779 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages were automatically installed and are no longer required: 2024-04-18T10:04:07.779 INFO:teuthology.orchestra.run.smithi145.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-18T10:04:07.780 INFO:teuthology.orchestra.run.smithi145.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-18T10:04:07.780 INFO:teuthology.orchestra.run.smithi145.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-18T10:04:07.780 INFO:teuthology.orchestra.run.smithi145.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-18T10:04:07.780 INFO:teuthology.orchestra.run.smithi145.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-18T10:04:07.780 INFO:teuthology.orchestra.run.smithi145.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-18T10:04:07.781 INFO:teuthology.orchestra.run.smithi145.stdout:The following additional packages will be installed: 2024-04-18T10:04:07.781 INFO:teuthology.orchestra.run.smithi145.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-18T10:04:07.781 INFO:teuthology.orchestra.run.smithi145.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-18T10:04:07.781 INFO:teuthology.orchestra.run.smithi145.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-18T10:04:07.781 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-18T10:04:07.781 INFO:teuthology.orchestra.run.smithi145.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-18T10:04:07.782 INFO:teuthology.orchestra.run.smithi145.stdout:Suggested packages: 2024-04-18T10:04:07.782 INFO:teuthology.orchestra.run.smithi145.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-18T10:04:07.782 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-18T10:04:07.782 INFO:teuthology.orchestra.run.smithi145.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-18T10:04:07.823 INFO:teuthology.orchestra.run.smithi145.stdout:The following NEW packages will be installed: 2024-04-18T10:04:07.823 INFO:teuthology.orchestra.run.smithi145.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-18T10:04:07.823 INFO:teuthology.orchestra.run.smithi145.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-18T10:04:07.824 INFO:teuthology.orchestra.run.smithi145.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-18T10:04:07.824 INFO:teuthology.orchestra.run.smithi145.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-18T10:04:07.824 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-18T10:04:07.824 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-18T10:04:07.824 INFO:teuthology.orchestra.run.smithi145.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-18T10:04:07.825 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages will be upgraded: 2024-04-18T10:04:07.825 INFO:teuthology.orchestra.run.smithi145.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-18T10:04:08.004 INFO:teuthology.orchestra.run.smithi145.stdout:4 upgraded, 39 newly installed, 0 to remove and 298 not upgraded. 2024-04-18T10:04:08.004 INFO:teuthology.orchestra.run.smithi145.stdout:Need to get 55.7 MB of archives. 2024-04-18T10:04:08.005 INFO:teuthology.orchestra.run.smithi145.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-18T10:04:08.005 INFO:teuthology.orchestra.run.smithi145.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-18T10:04:08.292 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174068 files and directories currently installed.) 2024-04-18T10:04:08.297 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-18T10:04:08.426 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-18T10:04:08.670 INFO:teuthology.orchestra.run.smithi145.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-18T10:04:08.693 INFO:teuthology.orchestra.run.smithi145.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-18T10:04:08.802 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-18T10:04:08.825 INFO:teuthology.orchestra.run.smithi145.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-18T10:04:08.826 INFO:teuthology.orchestra.run.smithi145.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-18T10:04:08.828 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-18T10:04:08.852 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-18T10:04:08.854 INFO:teuthology.orchestra.run.smithi145.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-18T10:04:08.857 INFO:teuthology.orchestra.run.smithi145.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-18T10:04:08.860 INFO:teuthology.orchestra.run.smithi145.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-18T10:04:08.861 INFO:teuthology.orchestra.run.smithi145.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-18T10:04:08.933 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-18T10:04:09.027 INFO:teuthology.orchestra.run.smithi145.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-18T10:04:09.277 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-18T10:04:09.324 INFO:teuthology.orchestra.run.smithi145.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-18T10:04:09.333 INFO:teuthology.orchestra.run.smithi145.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-18T10:04:09.338 INFO:teuthology.orchestra.run.smithi145.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-18T10:04:09.423 INFO:teuthology.orchestra.run.smithi145.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-18T10:04:09.684 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package bison. 2024-04-18T10:04:09.791 INFO:teuthology.orchestra.run.smithi062.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174068 files and directories currently installed.) 2024-04-18T10:04:09.797 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-18T10:04:09.852 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-18T10:04:10.237 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-18T10:04:10.265 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-18T10:04:10.303 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-18T10:04:10.534 INFO:teuthology.orchestra.run.smithi145.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-18T10:04:10.539 INFO:teuthology.orchestra.run.smithi145.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-18T10:04:10.539 INFO:teuthology.orchestra.run.smithi145.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-18T10:04:10.546 INFO:teuthology.orchestra.run.smithi145.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-18T10:04:10.546 INFO:teuthology.orchestra.run.smithi145.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-18T10:04:10.557 INFO:teuthology.orchestra.run.smithi145.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-18T10:04:10.648 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package dump. 2024-04-18T10:04:10.677 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-18T10:04:10.715 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking dump (0.4b47-3) ... 2024-04-18T10:04:10.724 INFO:teuthology.orchestra.run.smithi145.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-18T10:04:11.086 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package indent. 2024-04-18T10:04:11.104 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-18T10:04:11.145 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-18T10:04:11.351 INFO:teuthology.orchestra.run.smithi145.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-18T10:04:11.351 INFO:teuthology.orchestra.run.smithi145.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-18T10:04:11.406 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-18T10:04:11.423 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-18T10:04:11.464 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-18T10:04:11.559 INFO:teuthology.orchestra.run.smithi145.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-18T10:04:11.560 INFO:teuthology.orchestra.run.smithi145.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-18T10:04:11.615 INFO:teuthology.orchestra.run.smithi145.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-18T10:04:11.682 INFO:teuthology.orchestra.run.smithi145.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-18T10:04:11.690 INFO:teuthology.orchestra.run.smithi145.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-18T10:04:11.691 INFO:teuthology.orchestra.run.smithi145.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-18T10:04:11.692 INFO:teuthology.orchestra.run.smithi145.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-18T10:04:11.693 INFO:teuthology.orchestra.run.smithi145.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-18T10:04:11.843 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-18T10:04:11.872 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-18T10:04:11.883 INFO:teuthology.orchestra.run.smithi145.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-18T10:04:11.910 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-18T10:04:12.130 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-18T10:04:12.147 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-18T10:04:12.179 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-18T10:04:12.297 INFO:teuthology.orchestra.run.smithi145.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-18T10:04:12.567 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-18T10:04:12.595 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-18T10:04:12.633 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-18T10:04:12.886 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-18T10:04:12.902 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-18T10:04:12.944 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-18T10:04:13.121 INFO:teuthology.orchestra.run.smithi145.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-18T10:04:13.122 INFO:teuthology.orchestra.run.smithi145.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-18T10:04:13.122 INFO:teuthology.orchestra.run.smithi145.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-18T10:04:13.122 INFO:teuthology.orchestra.run.smithi145.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-18T10:04:13.156 INFO:teuthology.orchestra.run.smithi145.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-18T10:04:13.156 INFO:teuthology.orchestra.run.smithi145.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-18T10:04:13.186 INFO:teuthology.orchestra.run.smithi145.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-18T10:04:13.187 INFO:teuthology.orchestra.run.smithi145.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-18T10:04:13.230 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libjson-perl. 2024-04-18T10:04:13.259 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-18T10:04:13.305 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-18T10:04:13.590 INFO:teuthology.orchestra.run.smithi145.stdout:Preconfiguring packages ... 2024-04-18T10:04:13.675 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-18T10:04:13.704 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-18T10:04:13.748 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 55.7 MB in 5s (10.4 MB/s) 2024-04-18T10:04:13.750 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-18T10:04:14.068 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173995 files and directories currently installed.) 2024-04-18T10:04:14.074 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-18T10:04:14.220 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-18T10:04:14.852 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-18T10:04:15.487 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-18T10:04:15.515 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-18T10:04:15.553 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-18T10:04:15.830 INFO:teuthology.orchestra.run.smithi145.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-18T10:04:15.839 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-18T10:04:15.868 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-18T10:04:15.914 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-18T10:04:16.093 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package flex. 2024-04-18T10:04:16.193 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173995 files and directories currently installed.) 2024-04-18T10:04:16.199 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-18T10:04:16.242 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-18T10:04:16.409 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-18T10:04:16.437 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-18T10:04:16.483 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-18T10:04:16.767 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-18T10:04:16.905 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-18T10:04:16.933 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-18T10:04:16.971 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-18T10:04:16.989 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-18T10:04:17.341 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-18T10:04:17.369 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-18T10:04:17.401 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-18T10:04:17.416 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-18T10:04:17.819 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-18T10:04:17.847 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-18T10:04:17.910 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-18T10:04:17.938 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174068 files and directories currently installed.) 2024-04-18T10:04:17.944 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-18T10:04:18.089 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-18T10:04:18.247 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-18T10:04:18.275 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-18T10:04:18.313 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-18T10:04:18.490 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-18T10:04:18.621 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-18T10:04:18.658 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-18T10:04:18.686 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-18T10:04:18.741 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-18T10:04:18.957 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-18T10:04:19.027 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-18T10:04:19.055 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-18T10:04:19.102 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-18T10:04:19.304 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package bison. 2024-04-18T10:04:19.368 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174068 files and directories currently installed.) 2024-04-18T10:04:19.373 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-18T10:04:19.416 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-18T10:04:19.590 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-18T10:04:19.618 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-18T10:04:19.648 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-18T10:04:19.799 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-18T10:04:19.828 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-18T10:04:19.866 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-18T10:04:19.967 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package modemmanager. 2024-04-18T10:04:19.996 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-18T10:04:20.034 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-18T10:04:20.186 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package dump. 2024-04-18T10:04:20.214 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-18T10:04:20.273 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking dump (0.4b47-3) ... 2024-04-18T10:04:20.599 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package network-manager. 2024-04-18T10:04:20.628 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-18T10:04:20.648 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package indent. 2024-04-18T10:04:20.666 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-18T10:04:20.676 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-18T10:04:20.714 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-18T10:04:20.934 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-18T10:04:20.950 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-18T10:04:20.984 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-18T10:04:21.232 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package ppp. 2024-04-18T10:04:21.261 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-18T10:04:21.313 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-18T10:04:21.341 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-18T10:04:21.380 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-18T10:04:21.525 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-18T10:04:21.641 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-18T10:04:21.656 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-18T10:04:21.699 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-18T10:04:21.986 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-18T10:04:21.997 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package pptp-linux. 2024-04-18T10:04:22.014 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-18T10:04:22.025 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-18T10:04:22.044 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-18T10:04:22.063 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-18T10:04:22.272 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-18T10:04:22.288 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-18T10:04:22.321 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-18T10:04:22.341 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-18T10:04:22.369 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-18T10:04:22.407 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-18T10:04:22.524 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libjson-perl. 2024-04-18T10:04:22.541 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-18T10:04:22.583 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-18T10:04:22.777 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-18T10:04:22.806 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-18T10:04:22.852 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql-client-common (238) ... 2024-04-18T10:04:22.895 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-18T10:04:22.922 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-18T10:04:22.961 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-18T10:04:23.190 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-18T10:04:23.218 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-18T10:04:23.256 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-18T10:04:23.870 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql-common. 2024-04-18T10:04:23.898 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-18T10:04:23.956 INFO:teuthology.orchestra.run.smithi062.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-18T10:04:23.984 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql-common (238) ... 2024-04-18T10:04:24.444 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql-14. 2024-04-18T10:04:24.472 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-18T10:04:24.514 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-18T10:04:24.639 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-18T10:04:24.667 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-18T10:04:24.706 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-18T10:04:24.925 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-18T10:04:24.941 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-18T10:04:24.975 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-18T10:04:25.286 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-18T10:04:25.303 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-18T10:04:25.345 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-18T10:04:25.689 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-18T10:04:25.717 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-18T10:04:25.756 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-18T10:04:25.914 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql. 2024-04-18T10:04:25.943 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-18T10:04:25.972 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql (14+238) ... 2024-04-18T10:04:26.034 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-18T10:04:26.061 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-18T10:04:26.092 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-18T10:04:26.199 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql-client. 2024-04-18T10:04:26.216 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-18T10:04:26.257 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql-client (14+238) ... 2024-04-18T10:04:26.454 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-18T10:04:26.481 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-18T10:04:26.501 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-18T10:04:26.520 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-18T10:04:26.530 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-18T10:04:26.568 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-18T10:04:26.856 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-18T10:04:26.885 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-18T10:04:26.948 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-18T10:04:27.073 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package tcl8.6. 2024-04-18T10:04:27.101 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-18T10:04:27.209 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-18T10:04:27.225 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-18T10:04:27.259 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-18T10:04:27.325 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-18T10:04:27.503 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-18T10:04:27.519 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-18T10:04:27.561 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-18T10:04:27.605 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package tcl. 2024-04-18T10:04:27.607 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-18T10:04:27.643 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-18T10:04:27.993 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package wpasupplicant. 2024-04-18T10:04:28.022 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-18T10:04:28.049 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-18T10:04:28.059 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-18T10:04:28.076 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-18T10:04:28.115 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-18T10:04:28.414 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-18T10:04:28.427 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package modemmanager. 2024-04-18T10:04:28.443 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-18T10:04:28.455 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-18T10:04:28.480 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-18T10:04:28.485 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-18T10:04:28.858 INFO:teuthology.orchestra.run.smithi062.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-18T10:04:28.887 INFO:teuthology.orchestra.run.smithi062.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-18T10:04:28.933 INFO:teuthology.orchestra.run.smithi062.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-18T10:04:29.092 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package network-manager. 2024-04-18T10:04:29.120 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-18T10:04:29.167 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-18T10:04:29.239 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql-client-common (238) ... 2024-04-18T10:04:29.481 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-18T10:04:29.607 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-18T10:04:29.733 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-18T10:04:29.791 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ppp. 2024-04-18T10:04:29.819 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-18T10:04:29.842 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-18T10:04:29.944 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-18T10:04:30.053 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-18T10:04:30.095 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-18T10:04:30.446 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package pptp-linux. 2024-04-18T10:04:30.462 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-18T10:04:30.504 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-18T10:04:30.610 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-18T10:04:30.692 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-18T10:04:30.724 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-18T10:04:30.740 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-18T10:04:30.782 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-18T10:04:31.030 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-18T10:04:31.030 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-18T10:04:31.061 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-18T10:04:31.088 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-18T10:04:31.119 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql-client-common (238) ... 2024-04-18T10:04:31.414 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-18T10:04:31.442 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-18T10:04:31.481 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-18T10:04:31.487 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-18T10:04:31.613 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-18T10:04:31.747 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-18T10:04:31.873 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-18T10:04:31.974 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-18T10:04:32.075 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-18T10:04:32.103 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql-common. 2024-04-18T10:04:32.131 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-18T10:04:32.184 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-18T10:04:32.190 INFO:teuthology.orchestra.run.smithi145.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-18T10:04:32.217 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql-common (238) ... 2024-04-18T10:04:32.626 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql-14. 2024-04-18T10:04:32.654 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-18T10:04:32.694 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-18T10:04:32.696 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-18T10:04:33.467 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-18T10:04:33.568 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up dump (0.4b47-3) ... 2024-04-18T10:04:33.653 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-18T10:04:33.719 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-18T10:04:33.845 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-18T10:04:33.971 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-18T10:04:34.098 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-18T10:04:34.223 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-18T10:04:34.315 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-18T10:04:34.375 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-18T10:04:34.476 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-18T10:04:34.644 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-18T10:04:34.671 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql. 2024-04-18T10:04:34.700 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-18T10:04:34.738 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql (14+238) ... 2024-04-18T10:04:34.770 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-18T10:04:34.896 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-18T10:04:34.990 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql-client. 2024-04-18T10:04:35.006 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-18T10:04:35.048 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql-client (14+238) ... 2024-04-18T10:04:35.064 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql-client (14+238) ... 2024-04-18T10:04:35.190 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-18T10:04:35.276 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-18T10:04:35.292 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-18T10:04:35.334 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-18T10:04:35.634 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:04:35.635 INFO:teuthology.orchestra.run.smithi062.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-18T10:04:35.635 INFO:teuthology.orchestra.run.smithi062.stdout:which means they are currently configured by ifupdown: 2024-04-18T10:04:35.635 INFO:teuthology.orchestra.run.smithi062.stdout:- enp3s0f1 2024-04-18T10:04:35.635 INFO:teuthology.orchestra.run.smithi062.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-18T10:04:35.635 INFO:teuthology.orchestra.run.smithi062.stdout:remove their configuration from /etc/network/interfaces. 2024-04-18T10:04:35.635 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:04:35.662 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package tcl8.6. 2024-04-18T10:04:35.689 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-18T10:04:35.728 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-18T10:04:35.756 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-18T10:04:36.014 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package tcl. 2024-04-18T10:04:36.042 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-18T10:04:36.064 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-18T10:04:36.085 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-18T10:04:36.382 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-18T10:04:36.384 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package wpasupplicant. 2024-04-18T10:04:36.412 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-18T10:04:36.442 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-18T10:04:36.797 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-18T10:04:36.824 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-18T10:04:36.859 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-18T10:04:36.863 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-18T10:04:37.002 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-18T10:04:37.129 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-18T10:04:37.254 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql-common (238) ... 2024-04-18T10:04:37.266 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-18T10:04:37.294 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-18T10:04:37.341 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-18T10:04:37.637 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql-client-common (238) ... 2024-04-18T10:04:37.847 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-18T10:04:37.965 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-18T10:04:38.066 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-18T10:04:38.192 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-18T10:04:38.213 INFO:teuthology.orchestra.run.smithi062.stdout:Adding user postgres to group ssl-cert 2024-04-18T10:04:38.318 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-18T10:04:38.442 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:04:38.442 INFO:teuthology.orchestra.run.smithi062.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-18T10:04:38.453 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-18T10:04:38.638 INFO:teuthology.orchestra.run.smithi062.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-18T10:04:38.638 INFO:teuthology.orchestra.run.smithi062.stdout:Removing obsolete dictionary files: 2024-04-18T10:04:38.944 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-18T10:04:39.025 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-18T10:04:39.226 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-18T10:04:39.615 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-18T10:04:39.788 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-18T10:04:39.788 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-18T10:04:39.957 INFO:teuthology.orchestra.run.smithi062.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-18T10:04:39.969 INFO:teuthology.orchestra.run.smithi062.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-18T10:04:40.019 INFO:teuthology.orchestra.run.smithi062.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-18T10:04:40.019 INFO:teuthology.orchestra.run.smithi062.stdout:This user must also own the server process. 2024-04-18T10:04:40.020 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:04:40.020 INFO:teuthology.orchestra.run.smithi062.stdout:The database cluster will be initialized with locale "en_US". 2024-04-18T10:04:40.020 INFO:teuthology.orchestra.run.smithi062.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-18T10:04:40.020 INFO:teuthology.orchestra.run.smithi062.stdout:The default text search configuration will be set to "english". 2024-04-18T10:04:40.020 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:04:40.020 INFO:teuthology.orchestra.run.smithi062.stdout:Data page checksums are disabled. 2024-04-18T10:04:40.020 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:04:40.020 INFO:teuthology.orchestra.run.smithi062.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-18T10:04:40.021 INFO:teuthology.orchestra.run.smithi062.stdout:creating subdirectories ... ok 2024-04-18T10:04:40.021 INFO:teuthology.orchestra.run.smithi062.stdout:selecting dynamic shared memory implementation ... posix 2024-04-18T10:04:40.033 INFO:teuthology.orchestra.run.smithi062.stdout:selecting default max_connections ... 100 2024-04-18T10:04:40.054 INFO:teuthology.orchestra.run.smithi062.stdout:selecting default shared_buffers ... 128MB 2024-04-18T10:04:40.065 INFO:teuthology.orchestra.run.smithi062.stdout:selecting default time zone ... Etc/UTC 2024-04-18T10:04:40.066 INFO:teuthology.orchestra.run.smithi062.stdout:creating configuration files ... ok 2024-04-18T10:04:40.180 INFO:teuthology.orchestra.run.smithi062.stdout:running bootstrap script ... ok 2024-04-18T10:04:40.287 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-18T10:04:40.413 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-18T10:04:40.547 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-18T10:04:40.665 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-18T10:04:40.774 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-18T10:04:40.886 INFO:teuthology.orchestra.run.smithi062.stdout:performing post-bootstrap initialization ... ok 2024-04-18T10:04:40.892 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-18T10:04:41.018 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-18T10:04:41.152 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-18T10:04:41.611 INFO:teuthology.orchestra.run.smithi062.stdout:syncing data to disk ... ok 2024-04-18T10:04:41.882 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-18T10:04:41.985 INFO:teuthology.orchestra.run.smithi062.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-18T10:04:41.992 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up dump (0.4b47-3) ... 2024-04-18T10:04:42.086 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-18T10:04:42.152 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-18T10:04:42.286 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-18T10:04:42.412 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-18T10:04:42.538 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-18T10:04:42.664 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-18T10:04:42.754 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-18T10:04:42.816 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-18T10:04:42.917 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-18T10:04:43.093 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-18T10:04:43.228 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-18T10:04:43.354 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-18T10:04:43.522 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql-client (14+238) ... 2024-04-18T10:04:43.689 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-18T10:04:44.135 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:04:44.135 INFO:teuthology.orchestra.run.smithi145.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-18T10:04:44.136 INFO:teuthology.orchestra.run.smithi145.stdout:which means they are currently configured by ifupdown: 2024-04-18T10:04:44.136 INFO:teuthology.orchestra.run.smithi145.stdout:- enp3s0f1 2024-04-18T10:04:44.136 INFO:teuthology.orchestra.run.smithi145.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-18T10:04:44.136 INFO:teuthology.orchestra.run.smithi145.stdout:remove their configuration from /etc/network/interfaces. 2024-04-18T10:04:44.136 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:04:44.250 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-18T10:04:44.570 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-18T10:04:44.864 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-18T10:04:44.918 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-18T10:04:44.990 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-18T10:04:45.116 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up postgresql (14+238) ... 2024-04-18T10:04:45.242 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-18T10:04:45.368 INFO:teuthology.orchestra.run.smithi062.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-18T10:04:45.461 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-18T10:04:45.629 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-18T10:04:45.629 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-18T10:04:45.688 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-18T10:04:45.886 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-18T10:04:46.111 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql-common (238) ... 2024-04-18T10:04:46.112 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-18T10:04:46.205 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-18T10:04:46.499 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-18T10:04:46.651 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-18T10:04:47.233 INFO:teuthology.orchestra.run.smithi145.stdout:Adding user postgres to group ssl-cert 2024-04-18T10:04:47.465 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:04:47.465 INFO:teuthology.orchestra.run.smithi145.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-18T10:04:47.657 INFO:teuthology.orchestra.run.smithi145.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-18T10:04:47.658 INFO:teuthology.orchestra.run.smithi145.stdout:Removing obsolete dictionary files: 2024-04-18T10:04:48.073 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-18T10:04:48.164 INFO:teuthology.orchestra.run.smithi062.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-18T10:04:48.192 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-18T10:04:48.582 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-18T10:04:48.625 INFO:teuthology.orchestra.run.smithi062.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-18T10:04:48.631 DEBUG:teuthology.parallel:result is None 2024-04-18T10:04:48.907 INFO:teuthology.orchestra.run.smithi145.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-18T10:04:48.934 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-18T10:04:49.032 INFO:teuthology.orchestra.run.smithi145.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-18T10:04:49.032 INFO:teuthology.orchestra.run.smithi145.stdout:This user must also own the server process. 2024-04-18T10:04:49.032 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:04:49.032 INFO:teuthology.orchestra.run.smithi145.stdout:The database cluster will be initialized with locale "en_US". 2024-04-18T10:04:49.032 INFO:teuthology.orchestra.run.smithi145.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-18T10:04:49.032 INFO:teuthology.orchestra.run.smithi145.stdout:The default text search configuration will be set to "english". 2024-04-18T10:04:49.032 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:04:49.032 INFO:teuthology.orchestra.run.smithi145.stdout:Data page checksums are disabled. 2024-04-18T10:04:49.032 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:04:49.032 INFO:teuthology.orchestra.run.smithi145.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-18T10:04:49.033 INFO:teuthology.orchestra.run.smithi145.stdout:creating subdirectories ... ok 2024-04-18T10:04:49.033 INFO:teuthology.orchestra.run.smithi145.stdout:selecting dynamic shared memory implementation ... posix 2024-04-18T10:04:49.049 INFO:teuthology.orchestra.run.smithi145.stdout:selecting default max_connections ... 100 2024-04-18T10:04:49.071 INFO:teuthology.orchestra.run.smithi145.stdout:selecting default shared_buffers ... 128MB 2024-04-18T10:04:49.081 INFO:teuthology.orchestra.run.smithi145.stdout:selecting default time zone ... Etc/UTC 2024-04-18T10:04:49.082 INFO:teuthology.orchestra.run.smithi145.stdout:creating configuration files ... ok 2024-04-18T10:04:49.196 INFO:teuthology.orchestra.run.smithi145.stdout:running bootstrap script ... ok 2024-04-18T10:04:49.851 INFO:teuthology.orchestra.run.smithi145.stdout:performing post-bootstrap initialization ... ok 2024-04-18T10:04:50.556 INFO:teuthology.orchestra.run.smithi145.stdout:syncing data to disk ... ok 2024-04-18T10:04:50.927 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-18T10:04:53.664 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-18T10:04:53.814 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-18T10:04:53.932 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql (14+238) ... 2024-04-18T10:04:54.042 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-18T10:04:54.151 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-18T10:04:54.412 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-18T10:04:54.412 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-18T10:04:54.970 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-18T10:04:55.054 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-18T10:04:55.348 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-18T10:04:55.501 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-18T10:04:56.512 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-18T10:04:56.612 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-18T10:04:57.118 INFO:teuthology.orchestra.run.smithi145.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-18T10:04:57.124 DEBUG:teuthology.parallel:result is None 2024-04-18T10:04:57.125 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a6549451253d26216d353246c0fe6f6d80f3b9cf 2024-04-18T10:04:57.482 DEBUG:teuthology.orchestra.run.smithi062:> dpkg-query -W -f '${Version}' ceph 2024-04-18T10:04:57.502 INFO:teuthology.orchestra.run.smithi062.stdout:19.0.0-3115-ga6549451-1jammy 2024-04-18T10:04:57.503 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3115-ga6549451-1jammy 2024-04-18T10:04:57.503 INFO:teuthology.task.install:The correct ceph version 19.0.0-3115-ga6549451-1jammy is installed. 2024-04-18T10:04:57.505 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a6549451253d26216d353246c0fe6f6d80f3b9cf 2024-04-18T10:04:57.837 DEBUG:teuthology.orchestra.run.smithi145:> dpkg-query -W -f '${Version}' ceph 2024-04-18T10:04:57.856 INFO:teuthology.orchestra.run.smithi145.stdout:19.0.0-3115-ga6549451-1jammy 2024-04-18T10:04:57.856 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3115-ga6549451-1jammy 2024-04-18T10:04:57.856 INFO:teuthology.task.install:The correct ceph version 19.0.0-3115-ga6549451-1jammy is installed. 2024-04-18T10:04:57.858 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-18T10:04:57.858 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:04:57.858 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-18T10:04:57.874 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:04:57.874 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-18T10:04:57.910 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-18T10:04:57.910 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:04:57.910 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2024-04-18T10:04:57.931 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-18T10:04:57.987 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:04:57.987 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/daemon-helper 2024-04-18T10:04:58.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-18T10:04:58.053 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-18T10:04:58.053 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:04:58.053 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-18T10:04:58.067 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-18T10:04:58.123 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:04:58.123 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-18T10:04:58.136 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-18T10:04:58.189 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-18T10:04:58.189 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:04:58.189 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/stdin-killer 2024-04-18T10:04:58.203 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-18T10:04:58.262 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:04:58.262 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/stdin-killer 2024-04-18T10:04:58.275 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-18T10:04:58.336 INFO:teuthology.run_tasks:Running task ceph... 2024-04-18T10:04:58.424 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-18T10:04:58.424 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /var/log/ceph 2024-04-18T10:04:58.427 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 777 /var/log/ceph 2024-04-18T10:04:58.439 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-18T10:04:58.440 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-18T10:04:58.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-18T10:04:58.495 INFO:tasks.ceph:Creating extra log directories... 2024-04-18T10:04:58.495 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-18T10:04:58.540 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-18T10:04:58.557 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-18T10:04:58.558 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-18T10:04:58.558 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-18T10:04:58.558 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:04:58.558 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-18T10:04:58.599 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-18T10:04:58.672 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:04:58.672 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-18T10:04:58.678 DEBUG:teuthology.orchestra.run.smithi145:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-18T10:04:58.759 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-18T10:04:58.759 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-18T10:04:58.759 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928', 'branch': 'main', 'description': 'fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7660928', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'a6549451253d26216d353246c0fe6f6d80f3b9cf'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a6549451253d26216d353246c0fe6f6d80f3b9cf'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '08d35a8d8529783882dd092c73c0b27be41c4d86'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 950, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 7975, 'sha1': 'a6549451253d26216d353246c0fe6f6d80f3b9cf', 'sleep_before_teardown': 0, 'subset': '475/512', 'suite': 'fs', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_08d35a8d8529783882dd092c73c0b27be41c4d86/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '08d35a8d8529783882dd092c73c0b27be41c4d86', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi062.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPPVfMD4DHC87XmWZt3TofmRlDjTt43T/RfV4ZFvgP2nfzE+gBYIP68S+VzT0UuVvprZV3CcxGQOBax7qpzfhkI=', 'smithi145.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0dbhDmL1EjXSQwVknmV/Y3yOKu6mDKjuLciUf2po3nF29NJv8cXAwqEelT3Q3MjvvxUvBSEnsiWju2Ip+rIVs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'a6549451253d26216d353246c0fe6f6d80f3b9cf'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'a6549451253d26216d353246c0fe6f6d80f3b9cf', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_cephfs_shell']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-17_20:24:14', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-18T10:04:58.760 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-18T10:04:58.762 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-18T10:04:58.803 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/run/ceph 2024-04-18T10:04:58.808 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m0777 -- /var/run/ceph 2024-04-18T10:04:58.858 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:04:58.858 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2024-04-18T10:04:58.867 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-18T10:04:58.867 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 2024-04-18T10:04:58.915 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-18T10:04:58.915 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T10:04:58.916 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-04-18T10:04:58.916 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T10:04:58.916 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-04-18 09:58:39.633695897 +0000 2024-04-18T10:04:58.916 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-18 09:58:39.341698504 +0000 2024-04-18T10:04:58.916 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-18 09:58:39.341698504 +0000 2024-04-18T10:04:58.916 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2024-04-18T10:04:58.916 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-18T10:04:58.970 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-04-18T10:04:58.970 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-04-18T10:04:58.970 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.00044024 s, 1.2 MB/s 2024-04-18T10:04:58.971 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-18T10:04:59.020 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 2024-04-18T10:04:59.067 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-18T10:04:59.067 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T10:04:59.067 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-04-18T10:04:59.067 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T10:04:59.067 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-04-18 09:58:40.101691717 +0000 2024-04-18T10:04:59.067 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-18 09:58:39.845694003 +0000 2024-04-18T10:04:59.068 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-18 09:58:39.845694003 +0000 2024-04-18T10:04:59.068 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2024-04-18T10:04:59.068 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-18T10:04:59.121 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-04-18T10:04:59.121 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-04-18T10:04:59.121 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000442662 s, 1.2 MB/s 2024-04-18T10:04:59.122 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-18T10:04:59.172 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 2024-04-18T10:04:59.219 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-18T10:04:59.219 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T10:04:59.219 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 924 Links: 1 2024-04-18T10:04:59.219 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T10:04:59.219 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-04-18 09:58:40.637686928 +0000 2024-04-18T10:04:59.219 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-18 09:58:40.353689466 +0000 2024-04-18T10:04:59.219 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-18 09:58:40.353689466 +0000 2024-04-18T10:04:59.220 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2024-04-18T10:04:59.220 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-18T10:04:59.274 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-04-18T10:04:59.274 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-04-18T10:04:59.274 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.00042788 s, 1.2 MB/s 2024-04-18T10:04:59.275 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-18T10:04:59.324 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 2024-04-18T10:04:59.371 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-18T10:04:59.371 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T10:04:59.371 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 941 Links: 1 2024-04-18T10:04:59.371 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T10:04:59.371 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-04-18 09:58:41.133682498 +0000 2024-04-18T10:04:59.371 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-04-18 09:58:40.865684892 +0000 2024-04-18T10:04:59.371 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-04-18 09:58:40.865684892 +0000 2024-04-18T10:04:59.371 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: - 2024-04-18T10:04:59.371 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-18T10:04:59.425 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-04-18T10:04:59.425 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-04-18T10:04:59.425 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000284711 s, 1.8 MB/s 2024-04-18T10:04:59.426 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-18T10:04:59.476 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-18T10:04:59.476 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:04:59.476 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/scratch_devs of=/dev/stdout 2024-04-18T10:04:59.482 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-18T10:04:59.482 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_1 2024-04-18T10:04:59.531 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-18T10:04:59.531 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T10:04:59.531 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-04-18T10:04:59.531 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T10:04:59.531 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-04-18 09:58:38.732882811 +0000 2024-04-18T10:04:59.531 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-04-18 09:58:38.444885371 +0000 2024-04-18T10:04:59.531 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-04-18 09:58:38.444885371 +0000 2024-04-18T10:04:59.531 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2024-04-18T10:04:59.532 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-18T10:04:59.586 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-04-18T10:04:59.586 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-04-18T10:04:59.586 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000448394 s, 1.1 MB/s 2024-04-18T10:04:59.587 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-18T10:04:59.636 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_2 2024-04-18T10:04:59.684 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-18T10:04:59.684 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T10:04:59.684 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 900 Links: 1 2024-04-18T10:04:59.684 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T10:04:59.684 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-04-18 09:58:39.248878224 +0000 2024-04-18T10:04:59.684 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-04-18 09:58:38.964880749 +0000 2024-04-18T10:04:59.684 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-04-18 09:58:38.964880749 +0000 2024-04-18T10:04:59.684 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2024-04-18T10:04:59.685 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-18T10:04:59.737 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-04-18T10:04:59.738 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-04-18T10:04:59.738 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000428994 s, 1.2 MB/s 2024-04-18T10:04:59.739 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-18T10:04:59.788 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_3 2024-04-18T10:04:59.835 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-18T10:04:59.835 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T10:04:59.835 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 920 Links: 1 2024-04-18T10:04:59.835 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T10:04:59.835 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-04-18 09:58:39.744873815 +0000 2024-04-18T10:04:59.835 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-04-18 09:58:39.460876340 +0000 2024-04-18T10:04:59.835 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-04-18 09:58:39.460876340 +0000 2024-04-18T10:04:59.835 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2024-04-18T10:04:59.836 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-18T10:04:59.888 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-04-18T10:04:59.889 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-04-18T10:04:59.889 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000431297 s, 1.2 MB/s 2024-04-18T10:04:59.890 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-18T10:04:59.939 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_4 2024-04-18T10:04:59.986 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-18T10:04:59.986 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-18T10:04:59.987 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-04-18T10:04:59.987 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-18T10:04:59.987 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-04-18 09:58:40.284869014 +0000 2024-04-18T10:04:59.987 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-04-18 09:58:39.996871574 +0000 2024-04-18T10:04:59.987 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-04-18 09:58:39.996871574 +0000 2024-04-18T10:04:59.987 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2024-04-18T10:04:59.987 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-18T10:05:00.044 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-04-18T10:05:00.044 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-04-18T10:05:00.044 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000310068 s, 1.7 MB/s 2024-04-18T10:05:00.045 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-18T10:05:00.091 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-18T10:05:00.091 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi145.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-18T10:05:00.092 INFO:tasks.ceph:Generating config... 2024-04-18T10:05:00.093 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-18T10:05:00.093 INFO:tasks.ceph:[client] debug client = 20 2024-04-18T10:05:00.093 INFO:tasks.ceph:[client] debug ms = 1 2024-04-18T10:05:00.093 INFO:tasks.ceph:[client] pid file = 2024-04-18T10:05:00.093 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-18T10:05:00.093 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-18T10:05:00.093 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-18T10:05:00.093 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-18T10:05:00.093 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-18T10:05:00.093 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-18T10:05:00.093 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-18T10:05:00.093 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-18T10:05:00.093 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-18T10:05:00.093 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-18T10:05:00.094 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-18T10:05:00.094 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-18T10:05:00.094 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-18T10:05:00.094 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-18T10:05:00.094 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-18T10:05:00.094 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-18T10:05:00.094 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-18T10:05:00.094 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-18T10:05:00.094 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-18T10:05:00.094 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-18T10:05:00.094 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-18T10:05:00.094 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-18T10:05:00.095 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-18T10:05:00.096 INFO:tasks.ceph:Setting up mon.a... 2024-04-18T10:05:00.096 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-18T10:05:00.131 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/ceph.keyring 2024-04-18T10:05:00.157 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-18T10:05:00.235 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-18T10:05:00.290 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.62'), ('mon.b', '172.21.15.145'), ('mon.c', '[v2:172.21.15.145:3301,v1:172.21.15.145:6790]')] 2024-04-18T10:05:00.290 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.62,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.b': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-18T10:05:00.291 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:00.291 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-18T10:05:00.339 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.62 --add b 172.21.15.145 --addv c '[v2:172.21.15.145:3301,v1:172.21.15.145:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:00.426 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:00.426 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: generated fsid 26864971-6592-407b-b492-2c0e3b7ea567 2024-04-18T10:05:00.426 INFO:teuthology.orchestra.run.smithi062.stdout:setting min_mon_release = quincy 2024-04-18T10:05:00.427 INFO:teuthology.orchestra.run.smithi062.stdout:epoch 0 2024-04-18T10:05:00.427 INFO:teuthology.orchestra.run.smithi062.stdout:fsid 26864971-6592-407b-b492-2c0e3b7ea567 2024-04-18T10:05:00.427 INFO:teuthology.orchestra.run.smithi062.stdout:last_changed 2024-04-18T10:05:00.431468+0000 2024-04-18T10:05:00.427 INFO:teuthology.orchestra.run.smithi062.stdout:created 2024-04-18T10:05:00.431468+0000 2024-04-18T10:05:00.427 INFO:teuthology.orchestra.run.smithi062.stdout:min_mon_release 17 (quincy) 2024-04-18T10:05:00.427 INFO:teuthology.orchestra.run.smithi062.stdout:election_strategy: 1 2024-04-18T10:05:00.427 INFO:teuthology.orchestra.run.smithi062.stdout:0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-04-18T10:05:00.427 INFO:teuthology.orchestra.run.smithi062.stdout:1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-04-18T10:05:00.427 INFO:teuthology.orchestra.run.smithi062.stdout:2: [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] mon.c 2024-04-18T10:05:00.427 INFO:teuthology.orchestra.run.smithi062.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-18T10:05:00.428 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-18T10:05:00.475 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 26864971-6592-407b-b492-2c0e3b7ea567... 2024-04-18T10:05:00.475 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-18T10:05:00.520 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-18T10:05:00.549 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.550 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-18T10:05:00.551 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = "172.21.15.62,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = 26864971-6592-407b-b492-2c0e3b7ea567 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2024-04-18T10:05:00.552 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock profile = high_recovery_ops 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug trim objects = true 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.553 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: bdev async discard = True 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: bdev enable discard = True 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore allocator = bitmap 2024-04-18T10:05:00.554 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 1/20 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 1/20 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 4/10 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-04-18T10:05:00.555 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: debug client = 20 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2024-04-18T10:05:00.556 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: mon down mkfs grace = 300 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: mon op complaint time = 120 2024-04-18T10:05:00.557 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: client mount timeout = 600 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: debug client = 20 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = "" 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.b] 2024-04-18T10:05:00.558 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.c] 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.a] 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.b] 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout:[mds] 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds = 20 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: debug mds balancer = 20 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug frag = True 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: mds debug scatterstat = True 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: mds op complaint time = 180 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: mds verify scatter = True 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: osd op complaint time = 180 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: rados mon op timeout = 900 2024-04-18T10:05:00.559 INFO:teuthology.orchestra.run.smithi062.stdout: rados osd op timeout = 900 2024-04-18T10:05:00.560 INFO:teuthology.orchestra.run.smithi145.stdout:[global] 2024-04-18T10:05:00.560 INFO:teuthology.orchestra.run.smithi145.stdout: chdir = "" 2024-04-18T10:05:00.560 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-18T10:05:00.560 INFO:teuthology.orchestra.run.smithi145.stdout: auth supported = cephx 2024-04-18T10:05:00.560 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.560 INFO:teuthology.orchestra.run.smithi145.stdout: filestore xattr use omap = true 2024-04-18T10:05:00.560 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.560 INFO:teuthology.orchestra.run.smithi145.stdout: mon clock drift allowed = 1.000 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: osd crush chooseleaf type = 0 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: auth debug = true 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on old message = true 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on bug = true 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: mon pg warn max object skew = 0 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: # disable pg_autoscaler by default for new pools 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default size = 2 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow primary affinity = true 2024-04-18T10:05:00.561 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow pg remap = true 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on legacy crush tunables = false 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on crush straw calc version zero = false 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on no sortbitwise = false 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on osd down out interval zero = false 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on too few osds = false 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: mon_allow_pool_size_one = true 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: osd default data pool replay window = 5 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: mon allow pool delete = true 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.562 INFO:teuthology.orchestra.run.smithi145.stdout: mon cluster log file level = debug 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: debug asserts on shutdown = true 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: mon health detail to clog = false 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: mon host = "172.21.15.62,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: fsid = 26864971-6592-407b-b492-2c0e3b7ea567 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout:[osd] 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: osd journal size = 100 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub load threshold = 5.0 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub max interval = 600 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: osd mclock profile = high_recovery_ops 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: osd recover clone overlap = true 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: osd recovery max chunk = 1048576 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug shutdown = true 2024-04-18T10:05:00.563 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug op order = true 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify stray on activate = true 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug trim objects = true 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: osd open classes on start = true 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug pg log writeout = true 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: osd deep scrub update digest min age = 30 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: osd map max advance = 10 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: journal zero on create = true 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: filestore ondisk finisher threads = 3 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: filestore apply finisher threads = 3 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: bdev debug aio = true 2024-04-18T10:05:00.564 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug misdirected ops = true 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: bdev async discard = True 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: bdev enable discard = True 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore allocator = bitmap 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore block size = 96636764160 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore fsck on mount = True 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluefs = 1/20 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluestore = 1/20 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: debug osd = 20 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: debug rocksdb = 4/10 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd backfillfull_ratio = 0.85 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd full ratio = 0.9 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd nearfull ratio = 0.8 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: osd failsafe full ratio = 0.95 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: osd objectstore = bluestore 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-04-18T10:05:00.565 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.566 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr] 2024-04-18T10:05:00.566 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-04-18T10:05:00.566 INFO:teuthology.orchestra.run.smithi145.stdout: debug mgr = 20 2024-04-18T10:05:00.566 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-04-18T10:05:00.566 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-04-18T10:05:00.566 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-04-18T10:05:00.566 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-04-18T10:05:00.566 INFO:teuthology.orchestra.run.smithi145.stdout: mgr/telemetry/nag = false 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout:[mon] 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: debug paxos = 20 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: mon data avail warn = 5 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: mon mgr mkfs grace = 240 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd reporter subtree level = osd 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd prime pg temp = true 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.567 INFO:teuthology.orchestra.run.smithi145.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-18T10:05:00.568 INFO:teuthology.orchestra.run.smithi145.stdout: auth mon ticket ttl = 660 # 11m 2024-04-18T10:05:00.568 INFO:teuthology.orchestra.run.smithi145.stdout: auth service ticket ttl = 240 # 4m 2024-04-18T10:05:00.568 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.568 INFO:teuthology.orchestra.run.smithi145.stdout: # don't complain about insecure global_id in the test suite 2024-04-18T10:05:00.568 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-18T10:05:00.568 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-18T10:05:00.568 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.568 INFO:teuthology.orchestra.run.smithi145.stdout: # 1m isn't quite enough 2024-04-18T10:05:00.568 INFO:teuthology.orchestra.run.smithi145.stdout: mon_down_mkfs_grace = 2m 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_filestore_osds = false 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: mon down mkfs grace = 300 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: mon op complaint time = 120 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout:[client] 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: rgw cache enabled = true 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable ops log = true 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable usage log = true 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: client mount timeout = 600 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = "" 2024-04-18T10:05:00.569 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.a] 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.b] 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.c] 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.a] 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.b] 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout:[mds] 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds = 20 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds balancer = 20 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug frag = True 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug scatterstat = True 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout: mds op complaint time = 180 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout: mds verify scatter = True 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-04-18T10:05:00.570 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-04-18T10:05:00.571 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-18T10:05:00.571 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-18T10:05:00.663 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-18T10:05:00.663 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:00.663 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-18T10:05:00.669 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:00.669 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-18T10:05:00.715 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T10:05:00.715 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:00.715 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-18T10:05:00.715 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-18T10:05:00.779 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:00.779 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:00.828 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi145.front.sepia.ceph.com 2024-04-18T10:05:00.828 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:05:00.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-18T10:05:00.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-18T10:05:00.851 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:05:00.851 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:00.898 INFO:tasks.ceph:Setting up mon nodes... 2024-04-18T10:05:00.899 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-18T10:05:00.899 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-18T10:05:00.945 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-18T10:05:00.948 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-18T10:05:00.992 INFO:teuthology.orchestra.run.smithi145.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-18T10:05:01.022 INFO:tasks.ceph:Setting up mds nodes... 2024-04-18T10:05:01.022 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-18T10:05:01.067 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-18T10:05:01.070 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-18T10:05:01.083 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-18T10:05:01.125 INFO:teuthology.orchestra.run.smithi145.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-18T10:05:01.128 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-18T10:05:01.182 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-18T10:05:01.182 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-18T10:05:01.218 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-18T10:05:01.229 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-18T10:05:01.229 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi145.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-18T10:05:01.229 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-18T10:05:01.281 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-18T10:05:01.281 INFO:tasks.ceph:role: osd.0 2024-04-18T10:05:01.281 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T10:05:01.281 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-18T10:05:01.342 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T10:05:01.342 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T10:05:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T10:05:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T10:05:01.343 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T10:05:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-04-18T10:05:01.343 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T10:05:01.343 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T10:05:01.343 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T10:05:01.343 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T10:05:01.798 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-04-18T10:05:01.800 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-04-18T10:05:01.800 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-18T10:05:01.892 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-18T10:05:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T10:05:01.901 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T10:05:01.901 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-18T10:05:01.952 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-18T10:05:01.952 INFO:tasks.ceph:role: osd.1 2024-04-18T10:05:01.952 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T10:05:01.953 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-18T10:05:02.016 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T10:05:02.017 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T10:05:02.017 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T10:05:02.017 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T10:05:02.017 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T10:05:02.017 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-04-18T10:05:02.017 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T10:05:02.017 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T10:05:02.017 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T10:05:02.017 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T10:05:02.484 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-04-18T10:05:02.486 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-04-18T10:05:02.486 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-18T10:05:02.657 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-18T10:05:02.669 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T10:05:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T10:05:02.670 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-18T10:05:02.722 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-18T10:05:02.722 INFO:tasks.ceph:role: osd.2 2024-04-18T10:05:02.723 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T10:05:02.723 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-18T10:05:02.787 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T10:05:02.787 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T10:05:02.787 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T10:05:02.787 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T10:05:02.787 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T10:05:02.787 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-04-18T10:05:02.787 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T10:05:02.787 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T10:05:02.787 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T10:05:02.787 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T10:05:03.254 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-04-18T10:05:03.256 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-04-18T10:05:03.256 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-18T10:05:03.395 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-18T10:05:03.406 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T10:05:03.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T10:05:03.407 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-18T10:05:03.463 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-18T10:05:03.463 INFO:tasks.ceph:role: osd.3 2024-04-18T10:05:03.463 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T10:05:03.463 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-18T10:05:03.527 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T10:05:03.527 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T10:05:03.527 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T10:05:03.527 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T10:05:03.527 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T10:05:03.527 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-04-18T10:05:03.527 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T10:05:03.527 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T10:05:03.527 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T10:05:03.527 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T10:05:03.952 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-04-18T10:05:03.954 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-04-18T10:05:03.954 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-18T10:05:04.047 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-18T10:05:04.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T10:05:04.056 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T10:05:04.056 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:04.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:04.152+0000 7f01291377c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-18T10:05:04.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:04.152+0000 7f01291377c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-18T10:05:04.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:04.152+0000 7f01291377c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-18T10:05:04.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:04.152+0000 7f01291377c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-18T10:05:04.154 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:04.152+0000 7f01291377c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-18T10:05:04.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:04.156+0000 7f01291377c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-18T10:05:07.141 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-18T10:05:07.155 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:07.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:07.236+0000 7f164915d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-18T10:05:07.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:07.240+0000 7f164915d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-18T10:05:07.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:07.240+0000 7f164915d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-18T10:05:07.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:07.240+0000 7f164915d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-18T10:05:07.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:07.240+0000 7f164915d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-18T10:05:07.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:07.240+0000 7f164915d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-18T10:05:10.214 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-18T10:05:10.272 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:10.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:10.352+0000 7f9ba10ef7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-18T10:05:10.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:10.352+0000 7f9ba10ef7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-18T10:05:10.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:10.352+0000 7f9ba10ef7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-18T10:05:10.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:10.352+0000 7f9ba10ef7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-18T10:05:10.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:10.352+0000 7f9ba10ef7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-18T10:05:10.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:10.356+0000 7f9ba10ef7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-18T10:05:13.329 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-18T10:05:13.384 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:13.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:13.467+0000 7ff874db97c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-18T10:05:13.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:13.467+0000 7ff874db97c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-18T10:05:13.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:13.467+0000 7ff874db97c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-18T10:05:13.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:13.467+0000 7ff874db97c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-18T10:05:13.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:13.467+0000 7ff874db97c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-18T10:05:13.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:13.467+0000 7ff874db97c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-18T10:05:16.441 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-18T10:05:16.496 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-18T10:05:16.513 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-18T10:05:16.513 INFO:tasks.ceph:role: osd.4 2024-04-18T10:05:16.513 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi145.front.sepia.ceph.com 2024-04-18T10:05:16.513 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-18T10:05:16.573 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T10:05:16.573 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T10:05:16.573 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T10:05:16.574 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T10:05:16.574 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T10:05:16.574 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-04-18T10:05:16.574 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T10:05:16.574 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T10:05:16.574 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T10:05:16.574 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T10:05:17.020 INFO:teuthology.orchestra.run.smithi145.stdout:Discarding blocks...Done. 2024-04-18T10:05:17.022 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-04-18T10:05:17.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-18T10:05:17.119 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-18T10:05:17.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T10:05:17.129 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T10:05:17.130 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-18T10:05:17.180 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-18T10:05:17.180 INFO:tasks.ceph:role: osd.5 2024-04-18T10:05:17.180 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi145.front.sepia.ceph.com 2024-04-18T10:05:17.180 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-18T10:05:17.241 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T10:05:17.242 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T10:05:17.242 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T10:05:17.242 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T10:05:17.242 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T10:05:17.242 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-04-18T10:05:17.242 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T10:05:17.242 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T10:05:17.242 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T10:05:17.242 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T10:05:17.709 INFO:teuthology.orchestra.run.smithi145.stdout:Discarding blocks...Done. 2024-04-18T10:05:17.711 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-04-18T10:05:17.711 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-18T10:05:17.857 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-18T10:05:17.867 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T10:05:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T10:05:17.868 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-18T10:05:17.922 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-18T10:05:17.922 INFO:tasks.ceph:role: osd.6 2024-04-18T10:05:17.922 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi145.front.sepia.ceph.com 2024-04-18T10:05:17.922 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-18T10:05:17.986 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T10:05:17.986 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T10:05:17.986 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T10:05:17.986 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T10:05:17.986 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T10:05:17.986 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-04-18T10:05:17.986 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T10:05:17.986 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T10:05:17.986 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T10:05:17.986 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T10:05:18.445 INFO:teuthology.orchestra.run.smithi145.stdout:Discarding blocks...Done. 2024-04-18T10:05:18.447 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-04-18T10:05:18.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-18T10:05:18.571 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-18T10:05:18.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T10:05:18.583 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T10:05:18.583 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-18T10:05:18.638 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-18T10:05:18.638 INFO:tasks.ceph:role: osd.7 2024-04-18T10:05:18.638 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi145.front.sepia.ceph.com 2024-04-18T10:05:18.638 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-18T10:05:18.701 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-18T10:05:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-18T10:05:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-18T10:05:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-18T10:05:18.702 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-18T10:05:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-04-18T10:05:18.702 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-18T10:05:18.702 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-18T10:05:18.702 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-18T10:05:18.702 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-18T10:05:19.141 INFO:teuthology.orchestra.run.smithi145.stdout:Discarding blocks...Done. 2024-04-18T10:05:19.143 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-04-18T10:05:19.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-18T10:05:19.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-18T10:05:19.288 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /sbin/restorecon: command not found 2024-04-18T10:05:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T10:05:19.289 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:19.405 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:19.405+0000 7fc22ea8b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-18T10:05:19.406 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:19.409+0000 7fc22ea8b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-18T10:05:19.406 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:19.409+0000 7fc22ea8b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-18T10:05:19.406 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:19.409+0000 7fc22ea8b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-18T10:05:19.406 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:19.409+0000 7fc22ea8b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-18T10:05:19.407 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:19.409+0000 7fc22ea8b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-18T10:05:22.386 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-18T10:05:22.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:22.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:22.481+0000 7f7f460787c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-18T10:05:22.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:22.481+0000 7f7f460787c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-18T10:05:22.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:22.481+0000 7f7f460787c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-18T10:05:22.480 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:22.481+0000 7f7f460787c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-18T10:05:22.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:22.481+0000 7f7f460787c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-18T10:05:22.481 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:22.481+0000 7f7f460787c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-18T10:05:25.502 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-18T10:05:25.559 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:25.639 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:25.641+0000 7f2f136037c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-18T10:05:25.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:25.641+0000 7f2f136037c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-18T10:05:25.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:25.641+0000 7f2f136037c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-18T10:05:25.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:25.641+0000 7f2f136037c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-18T10:05:25.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:25.641+0000 7f2f136037c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-18T10:05:25.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:25.641+0000 7f2f136037c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-18T10:05:28.610 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-18T10:05:28.667 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:28.750 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:28.753+0000 7fd889a177c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-18T10:05:28.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:28.753+0000 7fd889a177c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-18T10:05:28.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:28.753+0000 7fd889a177c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-18T10:05:28.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:28.753+0000 7fd889a177c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-18T10:05:28.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:28.753+0000 7fd889a177c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-18T10:05:28.751 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:28.753+0000 7fd889a177c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-18T10:05:28.761 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-18T10:05:28.763 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-18T10:05:31.737 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-18T10:05:31.751 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-18T10:05:31.751 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:31.751 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-18T10:05:31.765 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:31.766 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-18T10:05:31.820 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:31.820 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-18T10:05:31.875 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:31.875 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-18T10:05:31.931 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:31.931 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-18T10:05:31.986 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:31.986 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-18T10:05:32.043 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:05:32.043 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-18T10:05:32.056 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:05:32.056 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-18T10:05:32.110 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:05:32.110 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-18T10:05:32.165 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:05:32.165 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-18T10:05:32.217 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:05:32.217 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-18T10:05:32.269 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:05:32.270 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-18T10:05:32.325 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:32.326 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-18T10:05:32.331 INFO:tasks.ceph:Adding keys to all mons... 2024-04-18T10:05:32.331 DEBUG:teuthology.orchestra.run.smithi062:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-18T10:05:32.376 DEBUG:teuthology.orchestra.run.smithi145:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-18T10:05:32.386 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.y] 2024-04-18T10:05:32.386 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBM8CBmkBCfOBAACzvtC9y+pLGAftyUQAmg2Q== 2024-04-18T10:05:32.386 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.a] 2024-04-18T10:05:32.386 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBN8CBmQwZLBBAAd+7JyRw7FcVA0o4nq9is7w== 2024-04-18T10:05:32.386 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.0] 2024-04-18T10:05:32.386 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBQ8CBmW55eCRAA800gMwDdNIxWAtzhDjuuMA== 2024-04-18T10:05:32.386 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.1] 2024-04-18T10:05:32.386 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBT8CBmj9V2DhAAonHttsgbVKgzZoht9NyVjA== 2024-04-18T10:05:32.386 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.2] 2024-04-18T10:05:32.386 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBW8CBm6CJcFRAADOM4tR+oWmS24E7lZQ0i/Q== 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.3] 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBZ8CBmt/YZHBAAniWnP+/mo8DmMHvG1jujlg== 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.x] 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBM8CBmS9qDOxAAV3nS9EtJXwkY2e1xQlhsvQ== 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout:[mds.b] 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBN8CBmG07RBxAAfwMp6QSjdomEaKVUwxaQqA== 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.4] 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBf8CBmpsGDGBAAj9fJH1NSwQT+EjebyYw04g== 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.5] 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBi8CBm5LD4HBAAtfN4fLfIAacGG87VNrDFaQ== 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.6] 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBl8CBmP4t3JhAAls1hDJHR8ldfkyMg9isu1Q== 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.7] 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBo8CBm4iwTLRAAKiomOOGDcB7xyrS1aezjYw== 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout:[client.0] 2024-04-18T10:05:32.387 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBN8CBmSFtIDRAAjl9lXubksjAuIJZdtH8x2Q== 2024-04-18T10:05:32.389 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr.y] 2024-04-18T10:05:32.389 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBM8CBmkBCfOBAACzvtC9y+pLGAftyUQAmg2Q== 2024-04-18T10:05:32.389 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.a] 2024-04-18T10:05:32.389 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBN8CBmQwZLBBAAd+7JyRw7FcVA0o4nq9is7w== 2024-04-18T10:05:32.389 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.0] 2024-04-18T10:05:32.389 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBQ8CBmW55eCRAA800gMwDdNIxWAtzhDjuuMA== 2024-04-18T10:05:32.389 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.1] 2024-04-18T10:05:32.389 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBT8CBmj9V2DhAAonHttsgbVKgzZoht9NyVjA== 2024-04-18T10:05:32.389 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.2] 2024-04-18T10:05:32.389 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBW8CBm6CJcFRAADOM4tR+oWmS24E7lZQ0i/Q== 2024-04-18T10:05:32.389 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.3] 2024-04-18T10:05:32.389 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBZ8CBmt/YZHBAAniWnP+/mo8DmMHvG1jujlg== 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr.x] 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBM8CBmS9qDOxAAV3nS9EtJXwkY2e1xQlhsvQ== 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.b] 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBN8CBmG07RBxAAfwMp6QSjdomEaKVUwxaQqA== 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.4] 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBf8CBmpsGDGBAAj9fJH1NSwQT+EjebyYw04g== 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.5] 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBi8CBm5LD4HBAAtfN4fLfIAacGG87VNrDFaQ== 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.6] 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBl8CBmP4t3JhAAls1hDJHR8ldfkyMg9isu1Q== 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.7] 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBo8CBm4iwTLRAAKiomOOGDcB7xyrS1aezjYw== 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout:[client.0] 2024-04-18T10:05:32.390 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBN8CBmSFtIDRAAjl9lXubksjAuIJZdtH8x2Q== 2024-04-18T10:05:32.391 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-18T10:05:32.432 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-18T10:05:32.477 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-18T10:05:32.479 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-18T10:05:32.565 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:32.567 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:32.607 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:32.652 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:32.699 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:32.740 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:32.783 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:32.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:32.869 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-18T10:05:32.912 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-18T10:05:32.955 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-18T10:05:32.957 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-18T10:05:33.043 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:33.045 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:33.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:33.132 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:33.174 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:33.220 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:33.265 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:33.304 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-18T10:05:33.345 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-18T10:05:33.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-18T10:05:33.439 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-18T10:05:33.439 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-18T10:05:33.484 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-18T10:05:33.577 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-18T10:05:33.588 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-18T10:05:33.602 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-18T10:05:33.698 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-18T10:05:33.708 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-18T10:05:33.760 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-18T10:05:33.848 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-18T10:05:33.901 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:33.903 DEBUG:teuthology.orchestra.run.smithi145:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-18T10:05:33.947 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-18T10:05:33.947 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-18T10:05:33.947 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-18T10:05:33.952 INFO:tasks.ceph.mon.a:Started 2024-04-18T10:05:33.952 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-18T10:05:33.952 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-18T10:05:33.992 INFO:tasks.ceph.mon.b:Started 2024-04-18T10:05:33.992 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-18T10:05:33.992 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-18T10:05:33.995 INFO:tasks.ceph.mon.c:Started 2024-04-18T10:05:33.995 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-18T10:05:33.995 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-18T10:05:33.995 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-18T10:05:33.997 INFO:tasks.ceph.mgr.y:Started 2024-04-18T10:05:33.997 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-18T10:05:33.998 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-18T10:05:34.000 INFO:tasks.ceph.mgr.x:Started 2024-04-18T10:05:34.000 INFO:tasks.ceph:Setting crush tunables to default 2024-04-18T10:05:34.000 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd crush tunables default 2024-04-18T10:05:34.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe71481640 1 Processor -- start 2024-04-18T10:05:34.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe71481640 1 -- start start 2024-04-18T10:05:34.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe71481640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 0x7fbe6c157b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:34.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe71481640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe6c0b5fb0 con 0x7fbe6c14f800 2024-04-18T10:05:34.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe71481640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe6c0b6eb0 con 0x7fbe6c09bd30 2024-04-18T10:05:34.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe71481640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fbe6c09c110 con 0x7fbe6c14fc60 2024-04-18T10:05:34.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe6a7fc640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbe6c14f800 0x7fbe6c1494a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50794/0 (socket says 172.21.15.62:50794) 2024-04-18T10:05:34.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe6a7fc640 1 -- 172.21.15.62:0/2002293702 learned_addr learned my addr 172.21.15.62:0/2002293702 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:05:34.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe6b7fe640 1 --2- 172.21.15.62:0/2002293702 >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 0x7fbe6c157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:34.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe6b7fe640 1 -- 172.21.15.62:0/2002293702 >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 msgr2=0x7fbe6c157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-18T10:05:34.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe6b7fe640 1 -- 172.21.15.62:0/2002293702 >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 msgr2=0x7fbe6c157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-18T10:05:34.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe6b7fe640 1 --2- 172.21.15.62:0/2002293702 >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 0x7fbe6c157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-18T10:05:34.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.119+0000 7fbe6b7fe640 1 --2- 172.21.15.62:0/2002293702 >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 0x7fbe6c157b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-18T10:05:34.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.323+0000 7fbe6b7fe640 1 --2- 172.21.15.62:0/2002293702 >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 0x7fbe6c157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:34.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.323+0000 7fbe6b7fe640 1 -- 172.21.15.62:0/2002293702 >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 msgr2=0x7fbe6c157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-18T10:05:34.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.323+0000 7fbe6b7fe640 1 -- 172.21.15.62:0/2002293702 >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 msgr2=0x7fbe6c157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-18T10:05:34.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.323+0000 7fbe6b7fe640 1 --2- 172.21.15.62:0/2002293702 >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 0x7fbe6c157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-18T10:05:34.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.323+0000 7fbe6b7fe640 1 --2- 172.21.15.62:0/2002293702 >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 0x7fbe6c157b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-18T10:05:34.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.363+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3586868600 0 0) 0x7fbe6c0b5fb0 con 0x7fbe6c14f800 2024-04-18T10:05:34.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.363+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe54003530 con 0x7fbe6c14f800 2024-04-18T10:05:34.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.363+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1970379425 0 0) 0x7fbe54003530 con 0x7fbe6c14f800 2024-04-18T10:05:34.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.363+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 msgr2=0x7fbe6c157b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-18T10:05:34.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.363+0000 7fbe69ffb640 1 --2- 172.21.15.62:0/2002293702 >> v2:172.21.15.62:3300/0 conn(0x7fbe6c14fc60 0x7fbe6c157b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.363+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 >> v1:172.21.15.145:6789/0 conn(0x7fbe6c09bd30 legacy=0x7fbe6c06bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:34.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.363+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6c0b63d0 con 0x7fbe6c14f800 2024-04-18T10:05:34.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.363+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fbe5c001680 con 0x7fbe6c14f800 2024-04-18T10:05:34.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.363+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 >> v1:172.21.15.62:6789/0 conn(0x7fbe6c14f800 legacy=0x7fbe6c1494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:34.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.363+0000 7fbe69ffb640 1 --2- 172.21.15.62:0/2002293702 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe54003a10 0x7fbe54003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:34.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.363+0000 7fbe69ffb640 1 --2- 172.21.15.62:0/2002293702 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe54004810 0x7fbe54004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:34.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.367+0000 7fbe69ffb640 1 --2- 172.21.15.62:0/2002293702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 0x7fbe5400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:34.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.367+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c09c110 con 0x7fbe540052f0 2024-04-18T10:05:34.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.367+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fbe5400c210 con 0x7fbe54003a10 2024-04-18T10:05:34.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.367+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fbe5400c410 con 0x7fbe54004810 2024-04-18T10:05:34.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.367+0000 7fbe6b7fe640 1 --2- 172.21.15.62:0/2002293702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 0x7fbe5400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:34.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.367+0000 7fbe6affd640 1 --2- 172.21.15.62:0/2002293702 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe54003a10 0x7fbe54003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:34.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.367+0000 7fbe6a7fc640 1 --2- 172.21.15.62:0/2002293702 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe54004810 0x7fbe54004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:34.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.367+0000 7fbe6b7fe640 1 -- 172.21.15.62:0/2002293702 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe54004810 msgr2=0x7fbe54004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:34.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.367+0000 7fbe6b7fe640 1 --2- 172.21.15.62:0/2002293702 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe54004810 0x7fbe54004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.367+0000 7fbe6b7fe640 1 -- 172.21.15.62:0/2002293702 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe54003a10 msgr2=0x7fbe54003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:34.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.367+0000 7fbe6b7fe640 1 --2- 172.21.15.62:0/2002293702 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe54003a10 0x7fbe54003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.367+0000 7fbe6b7fe640 1 -- 172.21.15.62:0/2002293702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe54003530 con 0x7fbe540052f0 2024-04-18T10:05:34.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.371+0000 7fbe6b7fe640 1 -- 172.21.15.62:0/2002293702 reap_dead start 2024-04-18T10:05:34.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.371+0000 7fbe6b7fe640 1 --2- 172.21.15.62:0/2002293702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 0x7fbe5400b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009a40 tx=0x7fbe6401d010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf33fde6e00a652 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:34.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.371+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe64009640 con 0x7fbe540052f0 2024-04-18T10:05:34.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.371+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe640ac640 con 0x7fbe540052f0 2024-04-18T10:05:34.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.371+0000 7fbe69ffb640 1 -- 172.21.15.62:0/2002293702 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c007e60 con 0x7fbe540052f0 2024-04-18T10:05:34.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.371+0000 7fbe71481640 1 -- 172.21.15.62:0/2002293702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 msgr2=0x7fbe5400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:34.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.371+0000 7fbe71481640 1 --2- 172.21.15.62:0/2002293702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 0x7fbe5400b9d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbe64009a40 tx=0x7fbe6401d010 comp rx=0 tx=0).stop 2024-04-18T10:05:34.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.371+0000 7fbe71481640 1 -- 172.21.15.62:0/2002293702 shutdown_connections 2024-04-18T10:05:34.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.371+0000 7fbe71481640 1 --2- 172.21.15.62:0/2002293702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 0x7fbe5400b9d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.371+0000 7fbe71481640 1 -- 172.21.15.62:0/2002293702 >> 172.21.15.62:0/2002293702 conn(0x7fbe6c09c360 msgr2=0x7fbe6c09c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:34.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.371+0000 7fbe71481640 1 -- 172.21.15.62:0/2002293702 shutdown_connections 2024-04-18T10:05:34.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.371+0000 7fbe71481640 1 -- 172.21.15.62:0/2002293702 wait complete. 2024-04-18T10:05:34.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe71481640 1 Processor -- start 2024-04-18T10:05:34.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe71481640 1 -- start start 2024-04-18T10:05:34.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe71481640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 0x7fbe6c1d5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:34.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe71481640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe6c1d6a10 0x7fbe6c1d7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:34.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe71481640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe6c1e4c00 0x7fbe6c1e5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:34.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe71481640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c06c9c0 con 0x7fbe540052f0 2024-04-18T10:05:34.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe71481640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6c06c840 con 0x7fbe6c1e4c00 2024-04-18T10:05:34.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe71481640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fbe6c06bad0 con 0x7fbe6c1d6a10 2024-04-18T10:05:34.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe6affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 0x7fbe6c1d5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:34.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe6affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 0x7fbe6c1d5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54360/0 (socket says 172.21.15.62:54360) 2024-04-18T10:05:34.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe6affd640 1 -- 172.21.15.62:0/633859686 learned_addr learned my addr 172.21.15.62:0/633859686 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:34.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe6a7fc640 1 --2- 172.21.15.62:0/633859686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe6c1d6a10 0x7fbe6c1d7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:34.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe6b7fe640 1 --2- 172.21.15.62:0/633859686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe6c1e4c00 0x7fbe6c1e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:34.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe6affd640 1 -- 172.21.15.62:0/633859686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe6c1d6a10 msgr2=0x7fbe6c1d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:34.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe6affd640 1 --2- 172.21.15.62:0/633859686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe6c1d6a10 0x7fbe6c1d7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe6affd640 1 -- 172.21.15.62:0/633859686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe6c1e4c00 msgr2=0x7fbe6c1e5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:34.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe6affd640 1 --2- 172.21.15.62:0/633859686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe6c1e4c00 0x7fbe6c1e5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe6affd640 1 -- 172.21.15.62:0/633859686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6c1d6f60 con 0x7fbe540052f0 2024-04-18T10:05:34.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe6b7fe640 1 --2- 172.21.15.62:0/633859686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe6c1e4c00 0x7fbe6c1e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:05:34.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe6affd640 1 --2- 172.21.15.62:0/633859686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 0x7fbe6c1d5970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000f510 tx=0x7fbe6000f910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:34.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe537fe640 1 -- 172.21.15.62:0/633859686 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe600052e0 con 0x7fbe540052f0 2024-04-18T10:05:34.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe537fe640 1 -- 172.21.15.62:0/633859686 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe60018070 con 0x7fbe540052f0 2024-04-18T10:05:34.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe71481640 1 -- 172.21.15.62:0/633859686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe6c09ec50 con 0x7fbe540052f0 2024-04-18T10:05:34.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe537fe640 1 -- 172.21.15.62:0/633859686 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60005480 con 0x7fbe540052f0 2024-04-18T10:05:34.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.375+0000 7fbe71481640 1 -- 172.21.15.62:0/633859686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe6c126240 con 0x7fbe540052f0 2024-04-18T10:05:34.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.379+0000 7fbe537fe640 1 -- 172.21.15.62:0/633859686 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbe60003800 con 0x7fbe540052f0 2024-04-18T10:05:34.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.379+0000 7fbe537fe640 1 -- 172.21.15.62:0/633859686 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbe6001dd10 con 0x7fbe540052f0 2024-04-18T10:05:34.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.379+0000 7fbe71481640 1 -- 172.21.15.62:0/633859686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe6c1d5eb0 con 0x7fbe540052f0 2024-04-18T10:05:34.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.379+0000 7fbe537fe640 1 -- 172.21.15.62:0/633859686 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fbe60003c80 con 0x7fbe540052f0 2024-04-18T10:05:34.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.447+0000 7fbe71481640 1 -- 172.21.15.62:0/633859686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fbe6c14a8d0 con 0x7fbe540052f0 2024-04-18T10:05:34.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.451+0000 7fbe537fe640 1 -- 172.21.15.62:0/633859686 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fbe6001d460 con 0x7fbe540052f0 2024-04-18T10:05:34.452 INFO:teuthology.orchestra.run.smithi062.stderr:adjusted tunables profile to default 2024-04-18T10:05:34.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.451+0000 7fbe71481640 1 -- 172.21.15.62:0/633859686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 msgr2=0x7fbe6c1d5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:34.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.451+0000 7fbe71481640 1 --2- 172.21.15.62:0/633859686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 0x7fbe6c1d5970 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000f510 tx=0x7fbe6000f910 comp rx=0 tx=0).stop 2024-04-18T10:05:34.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.451+0000 7fbe71481640 1 -- 172.21.15.62:0/633859686 shutdown_connections 2024-04-18T10:05:34.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.451+0000 7fbe71481640 1 --2- 172.21.15.62:0/633859686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe6c1e4c00 0x7fbe6c1e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.451+0000 7fbe71481640 1 --2- 172.21.15.62:0/633859686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe6c1d6a10 0x7fbe6c1d7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.451+0000 7fbe71481640 1 --2- 172.21.15.62:0/633859686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe540052f0 0x7fbe6c1d5970 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.451+0000 7fbe71481640 1 -- 172.21.15.62:0/633859686 >> 172.21.15.62:0/633859686 conn(0x7fbe6c09c360 msgr2=0x7fbe6c0b7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:34.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.451+0000 7fbe71481640 1 -- 172.21.15.62:0/633859686 shutdown_connections 2024-04-18T10:05:34.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:34.451+0000 7fbe71481640 1 -- 172.21.15.62:0/633859686 wait complete. 2024-04-18T10:05:34.470 INFO:tasks.ceph:check_enable_crimson: False 2024-04-18T10:05:34.471 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-18T10:05:34.471 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:34.471 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-18T10:05:34.484 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:34.484 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-18T10:05:34.543 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:34.543 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-18T10:05:34.598 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:05:34.598 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-18T10:05:34.614 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:34.612+0000 7f67ad67b180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-18T10:05:34.650 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:05:34.650 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-18T10:05:34.653 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:34.651+0000 7f225fe53180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-18T10:05:34.663 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:05:34.663 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-18T10:05:34.717 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:05:34.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-18T10:05:34.770 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-18T10:05:34.770 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-18T10:05:34.825 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50 0 2024-04-18T10:05:34.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.992+0000 7fd09e6a2640 1 Processor -- start 2024-04-18T10:05:34.993 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.992+0000 7fd09e6a2640 1 -- start start 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.992+0000 7fd09e6a2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd0980b1180 0x7fd0980b5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.992+0000 7fd09e6a2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09809ba50 con 0x7fd098144fc0 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.992+0000 7fd09e6a2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd098145a00 con 0x7fd0980b1560 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.992+0000 7fd09e6a2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fd09809aeb0 con 0x7fd0980b1180 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.992+0000 7fd0977fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fd0980b1560 0x7fd09809b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:60014/0 (socket says 172.21.15.145:60014) 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.992+0000 7fd0977fe640 1 -- 172.21.15.145:0/465242904 learned_addr learned my addr 172.21.15.145:0/465242904 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.992+0000 7fd097fff640 1 --2- 172.21.15.145:0/465242904 >> v2:172.21.15.62:3300/0 conn(0x7fd0980b1180 0x7fd0980b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.992+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1485883859 0 0) 0x7fd098145a00 con 0x7fd0980b1560 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd084003530 con 0x7fd0980b1560 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 <== mon.1 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 192791943 0 0) 0x7fd084003530 con 0x7fd0980b1560 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 >> v2:172.21.15.62:3300/0 conn(0x7fd0980b1180 msgr2=0x7fd0980b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 --2- 172.21.15.145:0/465242904 >> v2:172.21.15.62:3300/0 conn(0x7fd0980b1180 0x7fd0980b5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 >> v1:172.21.15.62:6789/0 conn(0x7fd098144fc0 legacy=0x7fd09814f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09809ab80 con 0x7fd0980b1560 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 <== mon.1 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fd080001680 con 0x7fd0980b1560 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 >> v1:172.21.15.145:6789/0 conn(0x7fd0980b1560 legacy=0x7fd09809b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0840039c0 0x7fd084003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd084004810 0x7fd084004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0840052f0 0x7fd08400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd09809aeb0 con 0x7fd084004810 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fd08400c1c0 con 0x7fd0840039c0 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd08400c3c0 con 0x7fd0840052f0 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd097fff640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0840039c0 0x7fd084003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd09cc3f640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0840052f0 0x7fd08400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd0977fe640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd084004810 0x7fd084004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd097fff640 1 -- 172.21.15.145:0/465242904 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0840052f0 msgr2=0x7fd08400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd097fff640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0840052f0 0x7fd08400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd097fff640 1 -- 172.21.15.145:0/465242904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd084004810 msgr2=0x7fd084004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:34.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd097fff640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd084004810 0x7fd084004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd097fff640 1 -- 172.21.15.145:0/465242904 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd084003530 con 0x7fd0840039c0 2024-04-18T10:05:34.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd09cc3f640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0840052f0 0x7fd08400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:05:34.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd0977fe640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd084004810 0x7fd084004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:05:34.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd09cc3f640 1 -- 172.21.15.145:0/465242904 reap_dead start 2024-04-18T10:05:34.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd097fff640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0840039c0 0x7fd084003de0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd090009ab0 tx=0x7fd09001c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=935880f0e194c6f server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:34.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900b1070 con 0x7fd0840039c0 2024-04-18T10:05:34.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0900a4950 con 0x7fd0840039c0 2024-04-18T10:05:34.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd096ffd640 1 -- 172.21.15.145:0/465242904 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900096b0 con 0x7fd0840039c0 2024-04-18T10:05:34.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd09e6a2640 1 -- 172.21.15.145:0/465242904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0840039c0 msgr2=0x7fd084003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:34.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd09e6a2640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0840039c0 0x7fd084003de0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd090009ab0 tx=0x7fd09001c710 comp rx=0 tx=0).stop 2024-04-18T10:05:34.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd09e6a2640 1 -- 172.21.15.145:0/465242904 shutdown_connections 2024-04-18T10:05:34.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd09e6a2640 1 --2- 172.21.15.145:0/465242904 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0840039c0 0x7fd084003de0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:34.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:34.996+0000 7fd09e6a2640 1 -- 172.21.15.145:0/465242904 >> 172.21.15.145:0/465242904 conn(0x7fd09809c360 msgr2=0x7fd09809c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:34.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09e6a2640 1 -- 172.21.15.145:0/465242904 shutdown_connections 2024-04-18T10:05:35.000 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09e6a2640 1 -- 172.21.15.145:0/465242904 wait complete. 2024-04-18T10:05:35.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09e6a2640 1 Processor -- start 2024-04-18T10:05:35.001 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09e6a2640 1 -- start start 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09e6a2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0840039c0 0x7fd0980b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09e6a2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0980b2ea0 0x7fd0981e0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09e6a2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0981e0ed0 0x7fd09813d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09e6a2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd098148ba0 con 0x7fd0840039c0 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09e6a2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fd098148a20 con 0x7fd0980b2ea0 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09e6a2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd098148d20 con 0x7fd0981e0ed0 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd0977fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0980b2ea0 0x7fd0981e0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09cc3f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0981e0ed0 0x7fd09813d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09cc3f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0981e0ed0 0x7fd09813d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:34580/0 (socket says 172.21.15.145:34580) 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09cc3f640 1 -- 172.21.15.145:0/186048663 learned_addr learned my addr 172.21.15.145:0/186048663 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd097fff640 1 --2- 172.21.15.145:0/186048663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0840039c0 0x7fd0980b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09cc3f640 1 -- 172.21.15.145:0/186048663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0980b2ea0 msgr2=0x7fd0981e0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.002 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09cc3f640 1 --2- 172.21.15.145:0/186048663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0980b2ea0 0x7fd0981e0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09cc3f640 1 -- 172.21.15.145:0/186048663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0840039c0 msgr2=0x7fd0980b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09cc3f640 1 --2- 172.21.15.145:0/186048663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0840039c0 0x7fd0980b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09cc3f640 1 -- 172.21.15.145:0/186048663 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0981e1330 con 0x7fd0981e0ed0 2024-04-18T10:05:35.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09cc3f640 1 --2- 172.21.15.145:0/186048663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0981e0ed0 0x7fd09813d340 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd0980b6870 tx=0x7fd08800ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:35.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd094ff9640 1 -- 172.21.15.145:0/186048663 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088001a90 con 0x7fd0981e0ed0 2024-04-18T10:05:35.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd094ff9640 1 -- 172.21.15.145:0/186048663 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd088018070 con 0x7fd0981e0ed0 2024-04-18T10:05:35.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09e6a2640 1 -- 172.21.15.145:0/186048663 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0980b42e0 con 0x7fd0981e0ed0 2024-04-18T10:05:35.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd094ff9640 1 -- 172.21.15.145:0/186048663 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08800d770 con 0x7fd0981e0ed0 2024-04-18T10:05:35.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd09e6a2640 1 -- 172.21.15.145:0/186048663 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0980b4870 con 0x7fd0981e0ed0 2024-04-18T10:05:35.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd094ff9640 1 -- 172.21.15.145:0/186048663 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd08800d910 con 0x7fd0981e0ed0 2024-04-18T10:05:35.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.000+0000 7fd094ff9640 1 -- 172.21.15.145:0/186048663 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fd08801d460 con 0x7fd0981e0ed0 2024-04-18T10:05:35.003 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.004+0000 7fd09e6a2640 1 -- 172.21.15.145:0/186048663 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0981e0670 con 0x7fd0981e0ed0 2024-04-18T10:05:35.015 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.016+0000 7fd094ff9640 1 -- 172.21.15.145:0/186048663 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd0981e0670 con 0x7fd0981e0ed0 2024-04-18T10:05:35.084 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.084+0000 7fd09e6a2640 1 -- 172.21.15.145:0/186048663 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50", "id": 0} v 0) -- 0x7fd098073f00 con 0x7fd0981e0ed0 2024-04-18T10:05:35.091 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.092+0000 7fd094ff9640 1 -- 172.21.15.145:0/186048663 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fd08800dd90 con 0x7fd0981e0ed0 2024-04-18T10:05:35.092 INFO:teuthology.orchestra.run.smithi145.stdout:0 2024-04-18T10:05:35.094 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.092+0000 7fd09e6a2640 1 -- 172.21.15.145:0/186048663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0981e0ed0 msgr2=0x7fd09813d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.094 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.092+0000 7fd09e6a2640 1 --2- 172.21.15.145:0/186048663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0981e0ed0 0x7fd09813d340 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd0980b6870 tx=0x7fd08800ff70 comp rx=0 tx=0).stop 2024-04-18T10:05:35.094 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.092+0000 7fd09e6a2640 1 -- 172.21.15.145:0/186048663 shutdown_connections 2024-04-18T10:05:35.094 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.092+0000 7fd09e6a2640 1 --2- 172.21.15.145:0/186048663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0981e0ed0 0x7fd09813d340 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.094 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.092+0000 7fd09e6a2640 1 --2- 172.21.15.145:0/186048663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0980b2ea0 0x7fd0981e0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.094 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.092+0000 7fd09e6a2640 1 --2- 172.21.15.145:0/186048663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0840039c0 0x7fd0980b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.094 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.092+0000 7fd09e6a2640 1 -- 172.21.15.145:0/186048663 >> 172.21.15.145:0/186048663 conn(0x7fd09809c360 msgr2=0x7fd098142a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:35.094 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.092+0000 7fd09e6a2640 1 -- 172.21.15.145:0/186048663 shutdown_connections 2024-04-18T10:05:35.094 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.092+0000 7fd09e6a2640 1 -- 172.21.15.145:0/186048663 wait complete. 2024-04-18T10:05:35.104 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:35.104+0000 7f67ad67b180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-18T10:05:35.115 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 71e116c5-995e-4a75-8218-2f5003e6a6a5 1 2024-04-18T10:05:35.218 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:35.215+0000 7f225fe53180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-18T10:05:35.229 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.228+0000 7f4526099640 1 Processor -- start 2024-04-18T10:05:35.229 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.228+0000 7f4526099640 1 -- start start 2024-04-18T10:05:35.230 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.228+0000 7f4526099640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f45200bb1d0 0x7f45201462b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.230 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.228+0000 7f4526099640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f452014e420 0x7f4520150810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.230 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.228+0000 7f4526099640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4520073800 con 0x7f4520073420 2024-04-18T10:05:35.230 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.228+0000 7f4526099640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f45200740d0 con 0x7f45200bb1d0 2024-04-18T10:05:35.230 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.228+0000 7f4526099640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f45200ba6f0 con 0x7f452014e420 2024-04-18T10:05:35.230 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.228+0000 7f451effd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f45200bb1d0 0x7f45201462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.231 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.228+0000 7f451ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f452014e420 0x7f4520150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.231 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.228+0000 7f451ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f452014e420 0x7f4520150810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:34596/0 (socket says 172.21.15.145:34596) 2024-04-18T10:05:35.231 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.228+0000 7f451effd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f45200bb1d0 0x7f45201462b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39138/0 (socket says 172.21.15.145:39138) 2024-04-18T10:05:35.231 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.228+0000 7f451ffff640 1 -- 172.21.15.145:0/2170292737 learned_addr learned my addr 172.21.15.145:0/2170292737 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-04-18T10:05:35.231 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f451ffff640 1 -- 172.21.15.145:0/2170292737 >> v2:172.21.15.145:3300/0 conn(0x7f45200bb1d0 msgr2=0x7f45201462b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:35.231 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f451ffff640 1 --2- 172.21.15.145:0/2170292737 >> v2:172.21.15.145:3300/0 conn(0x7f45200bb1d0 0x7f45201462b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.231 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f451ffff640 1 -- 172.21.15.145:0/2170292737 >> v1:172.21.15.62:6789/0 conn(0x7f4520073420 legacy=0x7f4520145ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.231 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f451ffff640 1 -- 172.21.15.145:0/2170292737 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4520073d60 con 0x7f452014e420 2024-04-18T10:05:35.231 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f451effd640 1 --2- 172.21.15.145:0/2170292737 >> v2:172.21.15.145:3300/0 conn(0x7f45200bb1d0 0x7f45201462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:05:35.232 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f451ffff640 1 --2- 172.21.15.145:0/2170292737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f452014e420 0x7f4520150810 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4518009e50 tx=0x7f451801c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc07ae2088566914 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:35.233 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f451e7fc640 1 -- 172.21.15.145:0/2170292737 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45180a56e0 con 0x7f452014e420 2024-04-18T10:05:35.233 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f451e7fc640 1 -- 172.21.15.145:0/2170292737 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45180a5880 con 0x7f452014e420 2024-04-18T10:05:35.233 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f451e7fc640 1 -- 172.21.15.145:0/2170292737 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45180a5c50 con 0x7f452014e420 2024-04-18T10:05:35.233 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f4526099640 1 -- 172.21.15.145:0/2170292737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f452014e420 msgr2=0x7f4520150810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.234 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f4526099640 1 --2- 172.21.15.145:0/2170292737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f452014e420 0x7f4520150810 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4518009e50 tx=0x7f451801c910 comp rx=0 tx=0).stop 2024-04-18T10:05:35.234 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f4526099640 1 -- 172.21.15.145:0/2170292737 shutdown_connections 2024-04-18T10:05:35.234 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f4526099640 1 --2- 172.21.15.145:0/2170292737 >> v2:172.21.15.145:3300/0 conn(0x7f45200bb1d0 0x7f45201462b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.234 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f4526099640 1 --2- 172.21.15.145:0/2170292737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f452014e420 0x7f4520150810 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.234 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f4526099640 1 -- 172.21.15.145:0/2170292737 >> 172.21.15.145:0/2170292737 conn(0x7f452009d470 msgr2=0x7f452009d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:35.234 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f4526099640 1 -- 172.21.15.145:0/2170292737 shutdown_connections 2024-04-18T10:05:35.234 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.232+0000 7f4526099640 1 -- 172.21.15.145:0/2170292737 wait complete. 2024-04-18T10:05:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4526099640 1 Processor -- start 2024-04-18T10:05:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4526099640 1 -- start start 2024-04-18T10:05:35.237 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4526099640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4520073420 0x7f452009e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.237 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4526099640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45200bb1d0 0x7f452009f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.237 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4526099640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f452014e420 0x7f45201debd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.237 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4526099640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f45201467f0 con 0x7f452014e420 2024-04-18T10:05:35.237 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4526099640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f4520070ec0 con 0x7f4520073420 2024-04-18T10:05:35.237 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4526099640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f4520146970 con 0x7f45200bb1d0 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f451f7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4520073420 0x7f452009e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f451effd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45200bb1d0 0x7f452009f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f451f7fe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4520073420 0x7f452009e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39144/0 (socket says 172.21.15.145:39144) 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f451effd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45200bb1d0 0x7f452009f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:34610/0 (socket says 172.21.15.145:34610) 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f451f7fe640 1 -- 172.21.15.145:0/2619824832 learned_addr learned my addr 172.21.15.145:0/2619824832 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f451ffff640 1 --2- 172.21.15.145:0/2619824832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f452014e420 0x7f45201debd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f451effd640 1 -- 172.21.15.145:0/2619824832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4520073420 msgr2=0x7f452009e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f451effd640 1 --2- 172.21.15.145:0/2619824832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4520073420 0x7f452009e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f451effd640 1 -- 172.21.15.145:0/2619824832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f452014e420 msgr2=0x7f45201debd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f451effd640 1 --2- 172.21.15.145:0/2619824832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f452014e420 0x7f45201debd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f451effd640 1 -- 172.21.15.145:0/2619824832 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f452009f810 con 0x7f45200bb1d0 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f451effd640 1 --2- 172.21.15.145:0/2619824832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45200bb1d0 0x7f452009f2d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f450800e8b0 tx=0x7f450800ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4517fff640 1 -- 172.21.15.145:0/2619824832 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4508017070 con 0x7f45200bb1d0 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4517fff640 1 -- 172.21.15.145:0/2619824832 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f450800c8e0 con 0x7f45200bb1d0 2024-04-18T10:05:35.238 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4526099640 1 -- 172.21.15.145:0/2619824832 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45201de6d0 con 0x7f45200bb1d0 2024-04-18T10:05:35.239 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4517fff640 1 -- 172.21.15.145:0/2619824832 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450800ccb0 con 0x7f45200bb1d0 2024-04-18T10:05:35.239 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4526099640 1 -- 172.21.15.145:0/2619824832 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f45200b18a0 con 0x7f45200bb1d0 2024-04-18T10:05:35.239 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4517fff640 1 -- 172.21.15.145:0/2619824832 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f450801a070 con 0x7f45200bb1d0 2024-04-18T10:05:35.239 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4517fff640 1 -- 172.21.15.145:0/2619824832 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f4508013470 con 0x7f45200bb1d0 2024-04-18T10:05:35.239 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.236+0000 7f4526099640 1 -- 172.21.15.145:0/2619824832 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4520069230 con 0x7f45200bb1d0 2024-04-18T10:05:35.240 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.240+0000 7f4517fff640 1 -- 172.21.15.145:0/2619824832 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f452009f810 con 0x7f45200bb1d0 2024-04-18T10:05:35.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.344+0000 7f4526099640 1 -- 172.21.15.145:0/2619824832 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd new", "uuid": "71e116c5-995e-4a75-8218-2f5003e6a6a5", "id": 1} v 0) -- 0x7f45200ba130 con 0x7f45200bb1d0 2024-04-18T10:05:35.347 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:35.348+0000 7f67ad67b180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-18T10:05:35.352 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.352+0000 7f4517fff640 1 -- 172.21.15.145:0/2619824832 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "71e116c5-995e-4a75-8218-2f5003e6a6a5", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f4508009320 con 0x7f45200bb1d0 2024-04-18T10:05:35.353 INFO:teuthology.orchestra.run.smithi145.stdout:1 2024-04-18T10:05:35.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.352+0000 7f4526099640 1 -- 172.21.15.145:0/2619824832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45200bb1d0 msgr2=0x7f452009f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.352+0000 7f4526099640 1 --2- 172.21.15.145:0/2619824832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45200bb1d0 0x7f452009f2d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f450800e8b0 tx=0x7f450800ed70 comp rx=0 tx=0).stop 2024-04-18T10:05:35.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.356+0000 7f4526099640 1 -- 172.21.15.145:0/2619824832 shutdown_connections 2024-04-18T10:05:35.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.356+0000 7f4526099640 1 --2- 172.21.15.145:0/2619824832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f452014e420 0x7f45201debd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.356+0000 7f4526099640 1 --2- 172.21.15.145:0/2619824832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45200bb1d0 0x7f452009f2d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.356+0000 7f4526099640 1 --2- 172.21.15.145:0/2619824832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4520073420 0x7f452009e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.356+0000 7f4526099640 1 -- 172.21.15.145:0/2619824832 >> 172.21.15.145:0/2619824832 conn(0x7f452009d470 msgr2=0x7f4520143aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:35.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.356+0000 7f4526099640 1 -- 172.21.15.145:0/2619824832 shutdown_connections 2024-04-18T10:05:35.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.356+0000 7f4526099640 1 -- 172.21.15.145:0/2619824832 wait complete. 2024-04-18T10:05:35.371 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new e6257209-4245-4fde-9aa9-461258b78577 2 2024-04-18T10:05:35.413 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:35.411+0000 7f225fe53180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-18T10:05:35.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.484+0000 7f53246b6640 1 Processor -- start 2024-04-18T10:05:35.485 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.484+0000 7f53246b6640 1 -- start start 2024-04-18T10:05:35.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.484+0000 7f53246b6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f531c0b1560 0x7f531c09b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.484+0000 7f53246b6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f531c144fc0 0x7f531c145410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f53246b6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f531c145a00 con 0x7f531c0b1180 2024-04-18T10:05:35.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f53246b6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f531c09aeb0 con 0x7f531c0b1560 2024-04-18T10:05:35.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f53246b6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f531c09ab80 con 0x7f531c144fc0 2024-04-18T10:05:35.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f5322c53640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f531c144fc0 0x7f531c145410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f5322c53640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f531c144fc0 0x7f531c145410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39160/0 (socket says 172.21.15.145:39160) 2024-04-18T10:05:35.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f5322c53640 1 -- 172.21.15.145:0/1923704637 learned_addr learned my addr 172.21.15.145:0/1923704637 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-04-18T10:05:35.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f5321c51640 1 --2- 172.21.15.145:0/1923704637 >> v2:172.21.15.62:3300/0 conn(0x7f531c0b1560 0x7f531c09b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.487 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f5322c53640 1 -- 172.21.15.145:0/1923704637 >> v2:172.21.15.62:3300/0 conn(0x7f531c0b1560 msgr2=0x7f531c09b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:35.488 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f5322c53640 1 --2- 172.21.15.145:0/1923704637 >> v2:172.21.15.62:3300/0 conn(0x7f531c0b1560 0x7f531c09b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.488 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f5322c53640 1 -- 172.21.15.145:0/1923704637 >> v1:172.21.15.62:6789/0 conn(0x7f531c0b1180 legacy=0x7f531c0b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:35.488 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f5322c53640 1 -- 172.21.15.145:0/1923704637 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531c0b4fb0 con 0x7f531c144fc0 2024-04-18T10:05:35.489 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f5322c53640 1 --2- 172.21.15.145:0/1923704637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f531c144fc0 0x7f531c145410 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f531c144a90 tx=0x7f531401c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e528d388a20fbc27 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:35.489 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f5321450640 1 -- 172.21.15.145:0/1923704637 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53140a5660 con 0x7f531c144fc0 2024-04-18T10:05:35.489 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f5321450640 1 -- 172.21.15.145:0/1923704637 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53140a5800 con 0x7f531c144fc0 2024-04-18T10:05:35.489 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f5321450640 1 -- 172.21.15.145:0/1923704637 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53140a5bd0 con 0x7f531c144fc0 2024-04-18T10:05:35.490 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f53246b6640 1 -- 172.21.15.145:0/1923704637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f531c144fc0 msgr2=0x7f531c145410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.490 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f53246b6640 1 --2- 172.21.15.145:0/1923704637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f531c144fc0 0x7f531c145410 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f531c144a90 tx=0x7f531401c890 comp rx=0 tx=0).stop 2024-04-18T10:05:35.490 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f53246b6640 1 -- 172.21.15.145:0/1923704637 shutdown_connections 2024-04-18T10:05:35.490 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f53246b6640 1 --2- 172.21.15.145:0/1923704637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f531c144fc0 0x7f531c145410 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.490 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f53246b6640 1 --2- 172.21.15.145:0/1923704637 >> v2:172.21.15.62:3300/0 conn(0x7f531c0b1560 0x7f531c09b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.490 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f53246b6640 1 -- 172.21.15.145:0/1923704637 >> 172.21.15.145:0/1923704637 conn(0x7f531c09c360 msgr2=0x7f531c09c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:35.490 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f53246b6640 1 -- 172.21.15.145:0/1923704637 shutdown_connections 2024-04-18T10:05:35.490 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.488+0000 7f53246b6640 1 -- 172.21.15.145:0/1923704637 wait complete. 2024-04-18T10:05:35.492 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f53246b6640 1 Processor -- start 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f53246b6640 1 -- start start 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f53246b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f531c0b1180 0x7f531c06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f53246b6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f531c0b1560 0x7f531c1de7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f53246b6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f531c144fc0 0x7f531c13f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f53246b6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f531c151ee0 con 0x7f531c144fc0 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f53246b6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f531c151d60 con 0x7f531c0b1560 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f53246b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f531c152060 con 0x7f531c0b1180 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5321c51640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f531c0b1560 0x7f531c1de7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5322452640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f531c0b1180 0x7f531c06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5321c51640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f531c0b1560 0x7f531c1de7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39168/0 (socket says 172.21.15.145:39168) 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5321c51640 1 -- 172.21.15.145:0/3204928694 learned_addr learned my addr 172.21.15.145:0/3204928694 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5322c53640 1 --2- 172.21.15.145:0/3204928694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f531c144fc0 0x7f531c13f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5322452640 1 -- 172.21.15.145:0/3204928694 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f531c0b1560 msgr2=0x7f531c1de7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5322452640 1 --2- 172.21.15.145:0/3204928694 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f531c0b1560 0x7f531c1de7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5322452640 1 -- 172.21.15.145:0/3204928694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f531c144fc0 msgr2=0x7f531c13f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5322452640 1 --2- 172.21.15.145:0/3204928694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f531c144fc0 0x7f531c13f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5322452640 1 -- 172.21.15.145:0/3204928694 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531c1e2030 con 0x7f531c0b1180 2024-04-18T10:05:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5321c51640 1 --2- 172.21.15.145:0/3204928694 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f531c0b1560 0x7f531c1de7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:05:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5322452640 1 --2- 172.21.15.145:0/3204928694 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f531c0b1180 0x7f531c06eab0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f531800f6d0 tx=0x7f531800fb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5312ffd640 1 -- 172.21.15.145:0/3204928694 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5318010690 con 0x7f531c0b1180 2024-04-18T10:05:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5312ffd640 1 -- 172.21.15.145:0/3204928694 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5318010830 con 0x7f531c0b1180 2024-04-18T10:05:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f53246b6640 1 -- 172.21.15.145:0/3204928694 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f531c1e2310 con 0x7f531c0b1180 2024-04-18T10:05:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5312ffd640 1 -- 172.21.15.145:0/3204928694 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5318010c60 con 0x7f531c0b1180 2024-04-18T10:05:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f53246b6640 1 -- 172.21.15.145:0/3204928694 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f531c1ded30 con 0x7f531c0b1180 2024-04-18T10:05:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5312ffd640 1 -- 172.21.15.145:0/3204928694 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f531800b450 con 0x7f531c0b1180 2024-04-18T10:05:35.494 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f5312ffd640 1 -- 172.21.15.145:0/3204928694 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f5318021a40 con 0x7f531c0b1180 2024-04-18T10:05:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.492+0000 7f53246b6640 1 -- 172.21.15.145:0/3204928694 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f531c069230 con 0x7f531c0b1180 2024-04-18T10:05:35.495 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.496+0000 7f5312ffd640 1 -- 172.21.15.145:0/3204928694 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5318010e00 con 0x7f531c0b1180 2024-04-18T10:05:35.564 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.564+0000 7f53246b6640 1 -- 172.21.15.145:0/3204928694 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e6257209-4245-4fde-9aa9-461258b78577", "id": 2} v 0) -- 0x7f531c0b93c0 con 0x7f531c0b1180 2024-04-18T10:05:35.570 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.572+0000 7f5312ffd640 1 -- 172.21.15.145:0/3204928694 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e6257209-4245-4fde-9aa9-461258b78577", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f5318021c40 con 0x7f531c0b1180 2024-04-18T10:05:35.570 INFO:teuthology.orchestra.run.smithi145.stdout:2 2024-04-18T10:05:35.571 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.572+0000 7f53246b6640 1 -- 172.21.15.145:0/3204928694 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f531c0b1180 msgr2=0x7f531c06eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.572 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.572+0000 7f53246b6640 1 --2- 172.21.15.145:0/3204928694 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f531c0b1180 0x7f531c06eab0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f531800f6d0 tx=0x7f531800fb90 comp rx=0 tx=0).stop 2024-04-18T10:05:35.572 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.572+0000 7f53246b6640 1 -- 172.21.15.145:0/3204928694 shutdown_connections 2024-04-18T10:05:35.572 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.572+0000 7f53246b6640 1 --2- 172.21.15.145:0/3204928694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f531c144fc0 0x7f531c13f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.572 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.572+0000 7f53246b6640 1 --2- 172.21.15.145:0/3204928694 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f531c0b1560 0x7f531c1de7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.572 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.572+0000 7f53246b6640 1 --2- 172.21.15.145:0/3204928694 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f531c0b1180 0x7f531c06eab0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.572 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.572+0000 7f53246b6640 1 -- 172.21.15.145:0/3204928694 >> 172.21.15.145:0/3204928694 conn(0x7f531c09c360 msgr2=0x7f531c1440f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:35.572 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.572+0000 7f53246b6640 1 -- 172.21.15.145:0/3204928694 shutdown_connections 2024-04-18T10:05:35.572 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.572+0000 7f53246b6640 1 -- 172.21.15.145:0/3204928694 wait complete. 2024-04-18T10:05:35.587 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 80fad71f-3ef3-46ab-a81c-104077404f09 3 2024-04-18T10:05:35.701 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5c88f640 1 Processor -- start 2024-04-18T10:05:35.701 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5c88f640 1 -- start start 2024-04-18T10:05:35.702 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5c88f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faf5806bd30 0x7faf5806e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.702 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5c88f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf5814be90 0x7faf581529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.702 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5c88f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf5806c110 con 0x7faf5806e8f0 2024-04-18T10:05:35.702 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5c88f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7faf5809d1d0 con 0x7faf5806bd30 2024-04-18T10:05:35.702 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5c88f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7faf580b0b10 con 0x7faf5814be90 2024-04-18T10:05:35.702 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5659c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faf5806bd30 0x7faf5806e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.703 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf56d9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf5814be90 0x7faf581529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.703 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5659c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faf5806bd30 0x7faf5806e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39184/0 (socket says 172.21.15.145:39184) 2024-04-18T10:05:35.703 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5659c640 1 -- 172.21.15.145:0/1816566080 learned_addr learned my addr 172.21.15.145:0/1816566080 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-04-18T10:05:35.703 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf55d9b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7faf5806e8f0 0x7faf5814b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:60018/0 (socket says 172.21.15.145:60018) 2024-04-18T10:05:35.703 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5659c640 1 -- 172.21.15.145:0/1816566080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf5814be90 msgr2=0x7faf581529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:35.703 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5659c640 1 --2- 172.21.15.145:0/1816566080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf5814be90 0x7faf581529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.703 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5659c640 1 -- 172.21.15.145:0/1816566080 >> v1:172.21.15.145:6789/0 conn(0x7faf5806e8f0 legacy=0x7faf5814b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.703 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf5659c640 1 -- 172.21.15.145:0/1816566080 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf580b00e0 con 0x7faf5806bd30 2024-04-18T10:05:35.703 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.700+0000 7faf56d9d640 1 --2- 172.21.15.145:0/1816566080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf5814be90 0x7faf581529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:05:35.704 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.704+0000 7faf5659c640 1 --2- 172.21.15.145:0/1816566080 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf5806bd30 0x7faf5806e3b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faf48009870 tx=0x7faf4801c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c95ef0644a7807f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:35.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.704+0000 7faf5559a640 1 -- 172.21.15.145:0/1816566080 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf480b0070 con 0x7faf5806bd30 2024-04-18T10:05:35.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.704+0000 7faf5559a640 1 -- 172.21.15.145:0/1816566080 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf480ab600 con 0x7faf5806bd30 2024-04-18T10:05:35.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.704+0000 7faf5559a640 1 -- 172.21.15.145:0/1816566080 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf480ab9d0 con 0x7faf5806bd30 2024-04-18T10:05:35.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.704+0000 7faf5c88f640 1 -- 172.21.15.145:0/1816566080 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf5806bd30 msgr2=0x7faf5806e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.705 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.704+0000 7faf5c88f640 1 --2- 172.21.15.145:0/1816566080 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf5806bd30 0x7faf5806e3b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7faf48009870 tx=0x7faf4801c440 comp rx=0 tx=0).stop 2024-04-18T10:05:35.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.704+0000 7faf5c88f640 1 -- 172.21.15.145:0/1816566080 shutdown_connections 2024-04-18T10:05:35.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.704+0000 7faf5c88f640 1 --2- 172.21.15.145:0/1816566080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf5814be90 0x7faf581529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.704+0000 7faf5c88f640 1 --2- 172.21.15.145:0/1816566080 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf5806bd30 0x7faf5806e3b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.704+0000 7faf5c88f640 1 -- 172.21.15.145:0/1816566080 >> 172.21.15.145:0/1816566080 conn(0x7faf580ba710 msgr2=0x7faf580bab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:35.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.704+0000 7faf5c88f640 1 -- 172.21.15.145:0/1816566080 shutdown_connections 2024-04-18T10:05:35.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.704+0000 7faf5c88f640 1 -- 172.21.15.145:0/1816566080 wait complete. 2024-04-18T10:05:35.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5c88f640 1 Processor -- start 2024-04-18T10:05:35.706 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5c88f640 1 -- start start 2024-04-18T10:05:35.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5c88f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf5806bd30 0x7faf581489b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5c88f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf5806e8f0 0x7faf580b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5c88f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf5814be90 0x7faf580b2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5c88f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7faf580afa80 con 0x7faf5806e8f0 2024-04-18T10:05:35.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5c88f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7faf580af720 con 0x7faf5806bd30 2024-04-18T10:05:35.708 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5c88f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7faf58155730 con 0x7faf5814be90 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf5806bd30 0x7faf581489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf56d9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf5814be90 0x7faf580b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5659c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf5806bd30 0x7faf581489b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39198/0 (socket says 172.21.15.145:39198) 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5659c640 1 -- 172.21.15.145:0/1263781244 learned_addr learned my addr 172.21.15.145:0/1263781244 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf55d9b640 1 --2- 172.21.15.145:0/1263781244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf5806e8f0 0x7faf580b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf56d9d640 1 -- 172.21.15.145:0/1263781244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf5806bd30 msgr2=0x7faf581489b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf56d9d640 1 --2- 172.21.15.145:0/1263781244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf5806bd30 0x7faf581489b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf56d9d640 1 -- 172.21.15.145:0/1263781244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf5806e8f0 msgr2=0x7faf580b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf56d9d640 1 --2- 172.21.15.145:0/1263781244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf5806e8f0 0x7faf580b2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf56d9d640 1 -- 172.21.15.145:0/1263781244 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5814b6d0 con 0x7faf5814be90 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5659c640 1 --2- 172.21.15.145:0/1263781244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf5806bd30 0x7faf581489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf56d9d640 1 --2- 172.21.15.145:0/1263781244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf5814be90 0x7faf580b2550 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faf5000efc0 tx=0x7faf5000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf46ffd640 1 -- 172.21.15.145:0/1263781244 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50018030 con 0x7faf5814be90 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf46ffd640 1 -- 172.21.15.145:0/1263781244 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf5000cd30 con 0x7faf5814be90 2024-04-18T10:05:35.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf46ffd640 1 -- 172.21.15.145:0/1263781244 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf50003bc0 con 0x7faf5814be90 2024-04-18T10:05:35.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5c88f640 1 -- 172.21.15.145:0/1263781244 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf58148ef0 con 0x7faf5814be90 2024-04-18T10:05:35.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5c88f640 1 -- 172.21.15.145:0/1263781244 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faf5814a630 con 0x7faf5814be90 2024-04-18T10:05:35.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf46ffd640 1 -- 172.21.15.145:0/1263781244 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7faf5001b070 con 0x7faf5814be90 2024-04-18T10:05:35.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf46ffd640 1 -- 172.21.15.145:0/1263781244 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7faf5001d8a0 con 0x7faf5814be90 2024-04-18T10:05:35.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.708+0000 7faf5c88f640 1 -- 172.21.15.145:0/1263781244 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf58069230 con 0x7faf5814be90 2024-04-18T10:05:35.711 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.712+0000 7faf46ffd640 1 -- 172.21.15.145:0/1263781244 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7faf50022070 con 0x7faf5814be90 2024-04-18T10:05:35.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.780+0000 7faf5c88f640 1 -- 172.21.15.145:0/1263781244 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd new", "uuid": "80fad71f-3ef3-46ab-a81c-104077404f09", "id": 3} v 0) -- 0x7faf5806eeb0 con 0x7faf5814be90 2024-04-18T10:05:35.785 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.784+0000 7faf46ffd640 1 -- 172.21.15.145:0/1263781244 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "80fad71f-3ef3-46ab-a81c-104077404f09", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7faf50010040 con 0x7faf5814be90 2024-04-18T10:05:35.785 INFO:teuthology.orchestra.run.smithi145.stdout:3 2024-04-18T10:05:35.786 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.784+0000 7faf5c88f640 1 -- 172.21.15.145:0/1263781244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf5814be90 msgr2=0x7faf580b2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.786 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.784+0000 7faf5c88f640 1 --2- 172.21.15.145:0/1263781244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf5814be90 0x7faf580b2550 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faf5000efc0 tx=0x7faf5000c490 comp rx=0 tx=0).stop 2024-04-18T10:05:35.786 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.784+0000 7faf5c88f640 1 -- 172.21.15.145:0/1263781244 shutdown_connections 2024-04-18T10:05:35.786 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.784+0000 7faf5c88f640 1 --2- 172.21.15.145:0/1263781244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faf5814be90 0x7faf580b2550 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.786 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.784+0000 7faf5c88f640 1 --2- 172.21.15.145:0/1263781244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf5806e8f0 0x7faf580b2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.787 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.784+0000 7faf5c88f640 1 --2- 172.21.15.145:0/1263781244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faf5806bd30 0x7faf581489b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.787 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.784+0000 7faf5c88f640 1 -- 172.21.15.145:0/1263781244 >> 172.21.15.145:0/1263781244 conn(0x7faf580ba710 msgr2=0x7faf5809c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:35.787 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.784+0000 7faf5c88f640 1 -- 172.21.15.145:0/1263781244 shutdown_connections 2024-04-18T10:05:35.787 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.784+0000 7faf5c88f640 1 -- 172.21.15.145:0/1263781244 wait complete. 2024-04-18T10:05:35.802 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 5a341da7-aaa2-4439-956a-6e9dc824aa38 4 2024-04-18T10:05:35.912 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:35.912+0000 7f67ad67b180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-18T10:05:35.921 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.920+0000 7fd9d8e9e640 1 Processor -- start 2024-04-18T10:05:35.922 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.920+0000 7fd9d8e9e640 1 -- start start 2024-04-18T10:05:35.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d8e9e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 0x7fd9d409b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d8e9e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d4145a00 con 0x7fd9d40b1180 2024-04-18T10:05:35.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d8e9e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d409ab80 con 0x7fd9d4144fc0 2024-04-18T10:05:35.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d8e9e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d409aeb0 con 0x7fd9d40b1560 2024-04-18T10:05:35.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d1d9b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 0x7fd9d409b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d2d9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fd9d4144fc0 0x7fd9d414f330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:60030/0 (socket says 172.21.15.145:60030) 2024-04-18T10:05:35.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d2d9d640 1 -- 172.21.15.145:0/2934358053 learned_addr learned my addr 172.21.15.145:0/2934358053 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-04-18T10:05:35.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d1d9b640 1 -- 172.21.15.145:0/2934358053 >> v1:172.21.15.145:6789/0 conn(0x7fd9d4144fc0 legacy=0x7fd9d414f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.923 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d1d9b640 1 -- 172.21.15.145:0/2934358053 >> v1:172.21.15.62:6789/0 conn(0x7fd9d40b1180 legacy=0x7fd9d40b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d159a640 1 -- 172.21.15.145:0/2934358053 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3499306906 0 0) 0x7fd9d409ab80 con 0x7fd9d4144fc0 2024-04-18T10:05:35.924 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d1d9b640 1 -- 172.21.15.145:0/2934358053 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d40b4fb0 con 0x7fd9d40b1560 2024-04-18T10:05:35.925 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d1d9b640 1 --2- 172.21.15.145:0/2934358053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 0x7fd9d409b340 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd9d409b030 tx=0x7fd9c801ea70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccdeb2a0adcb2687 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:35.925 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d159a640 1 -- 172.21.15.145:0/2934358053 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c80b4070 con 0x7fd9d40b1560 2024-04-18T10:05:35.925 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d159a640 1 -- 172.21.15.145:0/2934358053 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c80a7840 con 0x7fd9d40b1560 2024-04-18T10:05:35.926 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.924+0000 7fd9d159a640 1 -- 172.21.15.145:0/2934358053 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c80a7c10 con 0x7fd9d40b1560 2024-04-18T10:05:35.926 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/2934358053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 msgr2=0x7fd9d409b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.926 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 --2- 172.21.15.145:0/2934358053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 0x7fd9d409b340 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd9d409b030 tx=0x7fd9c801ea70 comp rx=0 tx=0).stop 2024-04-18T10:05:35.926 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/2934358053 shutdown_connections 2024-04-18T10:05:35.927 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 --2- 172.21.15.145:0/2934358053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 0x7fd9d409b340 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.927 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/2934358053 >> 172.21.15.145:0/2934358053 conn(0x7fd9d409c360 msgr2=0x7fd9d409c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:35.927 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/2934358053 shutdown_connections 2024-04-18T10:05:35.927 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/2934358053 wait complete. 2024-04-18T10:05:35.928 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 Processor -- start 2024-04-18T10:05:35.928 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 -- start start 2024-04-18T10:05:35.930 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9d40b1180 0x7fd9d41477b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.930 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 0x7fd9d4147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9d4144fc0 0x7fd9d41483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d406c670 con 0x7fd9d40b1180 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d406c4f0 con 0x7fd9d4144fc0 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d406c7f0 con 0x7fd9d40b1560 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d2d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9d4144fc0 0x7fd9d41483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d1d9b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 0x7fd9d4147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d1d9b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 0x7fd9d4147eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:34652/0 (socket says 172.21.15.145:34652) 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d2d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9d4144fc0 0x7fd9d41483f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39210/0 (socket says 172.21.15.145:39210) 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d1d9b640 1 -- 172.21.15.145:0/59727563 learned_addr learned my addr 172.21.15.145:0/59727563 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d259c640 1 --2- 172.21.15.145:0/59727563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9d40b1180 0x7fd9d41477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d2d9d640 1 -- 172.21.15.145:0/59727563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 msgr2=0x7fd9d4147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d2d9d640 1 --2- 172.21.15.145:0/59727563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 0x7fd9d4147eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d2d9d640 1 -- 172.21.15.145:0/59727563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9d40b1180 msgr2=0x7fd9d41477b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d2d9d640 1 --2- 172.21.15.145:0/59727563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9d40b1180 0x7fd9d41477b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d2d9d640 1 -- 172.21.15.145:0/59727563 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d4139ea0 con 0x7fd9d4144fc0 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d259c640 1 --2- 172.21.15.145:0/59727563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9d40b1180 0x7fd9d41477b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d1d9b640 1 --2- 172.21.15.145:0/59727563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 0x7fd9d4147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d2d9d640 1 --2- 172.21.15.145:0/59727563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9d4144fc0 0x7fd9d41483f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400c410 tx=0x7fd9c400c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9c2ffd640 1 -- 172.21.15.145:0/59727563 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c40074e0 con 0x7fd9d4144fc0 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9c2ffd640 1 -- 172.21.15.145:0/59727563 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c40047f0 con 0x7fd9d4144fc0 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/59727563 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d41396b0 con 0x7fd9d4144fc0 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9c2ffd640 1 -- 172.21.15.145:0/59727563 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4007680 con 0x7fd9d4144fc0 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/59727563 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d413a3d0 con 0x7fd9d4144fc0 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9c2ffd640 1 -- 172.21.15.145:0/59727563 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd9c4011070 con 0x7fd9d4144fc0 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.928+0000 7fd9c2ffd640 1 -- 172.21.15.145:0/59727563 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fd9c40138d0 con 0x7fd9d4144fc0 2024-04-18T10:05:35.932 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.932+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/59727563 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d4069230 con 0x7fd9d4144fc0 2024-04-18T10:05:35.933 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:35.932+0000 7fd9c2ffd640 1 -- 172.21.15.145:0/59727563 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd9c401edc0 con 0x7fd9d4144fc0 2024-04-18T10:05:35.969 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:35.968+0000 7f67ad67b180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-18T10:05:35.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.000+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/59727563 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5a341da7-aaa2-4439-956a-6e9dc824aa38", "id": 4} v 0) -- 0x7fd9d4073f00 con 0x7fd9d4144fc0 2024-04-18T10:05:36.007 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.008+0000 7fd9c2ffd640 1 -- 172.21.15.145:0/59727563 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5a341da7-aaa2-4439-956a-6e9dc824aa38", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fd9c4004da0 con 0x7fd9d4144fc0 2024-04-18T10:05:36.007 INFO:teuthology.orchestra.run.smithi145.stdout:4 2024-04-18T10:05:36.008 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.008+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/59727563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9d4144fc0 msgr2=0x7fd9d41483f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.008 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.008+0000 7fd9d8e9e640 1 --2- 172.21.15.145:0/59727563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9d4144fc0 0x7fd9d41483f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400c410 tx=0x7fd9c400c8d0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.008 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.008+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/59727563 shutdown_connections 2024-04-18T10:05:36.009 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.008+0000 7fd9d8e9e640 1 --2- 172.21.15.145:0/59727563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9d4144fc0 0x7fd9d41483f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.009 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.008+0000 7fd9d8e9e640 1 --2- 172.21.15.145:0/59727563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9d40b1560 0x7fd9d4147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.009 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.008+0000 7fd9d8e9e640 1 --2- 172.21.15.145:0/59727563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9d40b1180 0x7fd9d41477b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.009 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.008+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/59727563 >> 172.21.15.145:0/59727563 conn(0x7fd9d409c360 msgr2=0x7fd9d4144180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:36.009 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.008+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/59727563 shutdown_connections 2024-04-18T10:05:36.009 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.008+0000 7fd9d8e9e640 1 -- 172.21.15.145:0/59727563 wait complete. 2024-04-18T10:05:36.024 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9 5 2024-04-18T10:05:36.025 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:36.024+0000 7f67ad67b180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-18T10:05:36.039 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:36.039+0000 7f225fe53180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-18T10:05:36.095 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:36.095+0000 7f225fe53180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-18T10:05:36.104 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:36.104+0000 7f67ad67b180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-18T10:05:36.137 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.136+0000 7fa95b388640 1 Processor -- start 2024-04-18T10:05:36.137 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.136+0000 7fa95b388640 1 -- start start 2024-04-18T10:05:36.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.136+0000 7fa95b388640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa9540ba9a0 0x7fa954145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.136+0000 7fa95b388640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9540bb1b0 con 0x7fa95413fa20 2024-04-18T10:05:36.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.136+0000 7fa95b388640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9540bb490 con 0x7fa9540ba5c0 2024-04-18T10:05:36.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.136+0000 7fa95b388640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fa9540b96f0 con 0x7fa9540ba9a0 2024-04-18T10:05:36.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa958923640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa9540ba9a0 0x7fa954145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa959124640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fa9540ba5c0 0x7fa9540732e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:60040/0 (socket says 172.21.15.145:60040) 2024-04-18T10:05:36.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa959124640 1 -- 172.21.15.145:0/1159427385 learned_addr learned my addr 172.21.15.145:0/1159427385 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-04-18T10:05:36.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa958923640 1 -- 172.21.15.145:0/1159427385 >> v1:172.21.15.145:6789/0 conn(0x7fa9540ba5c0 legacy=0x7fa9540732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa958923640 1 -- 172.21.15.145:0/1159427385 >> v1:172.21.15.62:6789/0 conn(0x7fa95413fa20 legacy=0x7fa954150410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:36.139 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa958923640 1 -- 172.21.15.145:0/1159427385 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9541460b0 con 0x7fa9540ba9a0 2024-04-18T10:05:36.141 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa958923640 1 --2- 172.21.15.145:0/1159427385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9540ba9a0 0x7fa954145b70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa9440097c0 tx=0x7fa94401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5761ce8fcefac5db server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:36.141 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa94bfff640 1 -- 172.21.15.145:0/1159427385 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9440b0070 con 0x7fa9540ba9a0 2024-04-18T10:05:36.141 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa94bfff640 1 -- 172.21.15.145:0/1159427385 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9440ab600 con 0x7fa9540ba9a0 2024-04-18T10:05:36.141 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa94bfff640 1 -- 172.21.15.145:0/1159427385 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9440ab9d0 con 0x7fa9540ba9a0 2024-04-18T10:05:36.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa95b388640 1 -- 172.21.15.145:0/1159427385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9540ba9a0 msgr2=0x7fa954145b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa95b388640 1 --2- 172.21.15.145:0/1159427385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9540ba9a0 0x7fa954145b70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa9440097c0 tx=0x7fa94401c410 comp rx=0 tx=0).stop 2024-04-18T10:05:36.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa95b388640 1 -- 172.21.15.145:0/1159427385 shutdown_connections 2024-04-18T10:05:36.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa95b388640 1 --2- 172.21.15.145:0/1159427385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9540ba9a0 0x7fa954145b70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa95b388640 1 -- 172.21.15.145:0/1159427385 >> 172.21.15.145:0/1159427385 conn(0x7fa9540a0f00 msgr2=0x7fa95409aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:36.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa95b388640 1 -- 172.21.15.145:0/1159427385 shutdown_connections 2024-04-18T10:05:36.142 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.140+0000 7fa95b388640 1 -- 172.21.15.145:0/1159427385 wait complete. 2024-04-18T10:05:36.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa95b388640 1 Processor -- start 2024-04-18T10:05:36.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa95b388640 1 -- start start 2024-04-18T10:05:36.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa95b388640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9540ba5c0 0x7fa95406f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa95b388640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9540ba9a0 0x7fa95406e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa95b388640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95413fa20 0x7fa95409f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa95b388640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa9540bb800 con 0x7fa95413fa20 2024-04-18T10:05:36.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa95b388640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fa9540bbea0 con 0x7fa9540ba5c0 2024-04-18T10:05:36.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa95b388640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fa954154970 con 0x7fa9540ba9a0 2024-04-18T10:05:36.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa959124640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9540ba5c0 0x7fa95406f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.145 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa959124640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9540ba5c0 0x7fa95406f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39224/0 (socket says 172.21.15.145:39224) 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa958923640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9540ba9a0 0x7fa95406e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa959124640 1 -- 172.21.15.145:0/2633234420 learned_addr learned my addr 172.21.15.145:0/2633234420 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa959925640 1 --2- 172.21.15.145:0/2633234420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95413fa20 0x7fa95409f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa959124640 1 -- 172.21.15.145:0/2633234420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9540ba9a0 msgr2=0x7fa95406e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa959124640 1 --2- 172.21.15.145:0/2633234420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9540ba9a0 0x7fa95406e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa959124640 1 -- 172.21.15.145:0/2633234420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95413fa20 msgr2=0x7fa95409f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa959124640 1 --2- 172.21.15.145:0/2633234420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95413fa20 0x7fa95409f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa959124640 1 -- 172.21.15.145:0/2633234420 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95409ea50 con 0x7fa9540ba5c0 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa958923640 1 --2- 172.21.15.145:0/2633234420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9540ba9a0 0x7fa95406e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa959925640 1 --2- 172.21.15.145:0/2633234420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95413fa20 0x7fa95409f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa959124640 1 --2- 172.21.15.145:0/2633234420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9540ba5c0 0x7fa95406f900 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa95000c500 tx=0x7fa95000c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa949ffb640 1 -- 172.21.15.145:0/2633234420 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa950017070 con 0x7fa9540ba5c0 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa949ffb640 1 -- 172.21.15.145:0/2633234420 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa950005cf0 con 0x7fa9540ba5c0 2024-04-18T10:05:36.146 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa949ffb640 1 -- 172.21.15.145:0/2633234420 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9500072f0 con 0x7fa9540ba5c0 2024-04-18T10:05:36.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa95b388640 1 -- 172.21.15.145:0/2633234420 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa95406df80 con 0x7fa9540ba5c0 2024-04-18T10:05:36.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa95b388640 1 -- 172.21.15.145:0/2633234420 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa95409f3f0 con 0x7fa9540ba5c0 2024-04-18T10:05:36.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa949ffb640 1 -- 172.21.15.145:0/2633234420 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa95001a070 con 0x7fa9540ba5c0 2024-04-18T10:05:36.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa949ffb640 1 -- 172.21.15.145:0/2633234420 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fa95001ccb0 con 0x7fa9540ba5c0 2024-04-18T10:05:36.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.144+0000 7fa95b388640 1 -- 172.21.15.145:0/2633234420 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa954069230 con 0x7fa9540ba5c0 2024-04-18T10:05:36.147 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.148+0000 7fa949ffb640 1 -- 172.21.15.145:0/2633234420 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa9500074e0 con 0x7fa9540ba5c0 2024-04-18T10:05:36.150 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:36.151+0000 7f225fe53180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-18T10:05:36.159 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:36.160+0000 7f67ad67b180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-18T10:05:36.215 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.216+0000 7fa95b388640 1 -- 172.21.15.145:0/2633234420 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9", "id": 5} v 0) -- 0x7fa954144f40 con 0x7fa9540ba5c0 2024-04-18T10:05:36.222 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.220+0000 7fa949ffb640 1 -- 172.21.15.145:0/2633234420 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fa95001c440 con 0x7fa9540ba5c0 2024-04-18T10:05:36.222 INFO:teuthology.orchestra.run.smithi145.stdout:5 2024-04-18T10:05:36.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.224+0000 7fa95b388640 1 -- 172.21.15.145:0/2633234420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9540ba5c0 msgr2=0x7fa95406f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.224+0000 7fa95b388640 1 --2- 172.21.15.145:0/2633234420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9540ba5c0 0x7fa95406f900 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa95000c500 tx=0x7fa95000c9c0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.224+0000 7fa95b388640 1 -- 172.21.15.145:0/2633234420 shutdown_connections 2024-04-18T10:05:36.223 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.224+0000 7fa95b388640 1 --2- 172.21.15.145:0/2633234420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa95413fa20 0x7fa95409f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.224+0000 7fa95b388640 1 --2- 172.21.15.145:0/2633234420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9540ba9a0 0x7fa95406e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.224+0000 7fa95b388640 1 --2- 172.21.15.145:0/2633234420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa9540ba5c0 0x7fa95406f900 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.224+0000 7fa95b388640 1 -- 172.21.15.145:0/2633234420 >> 172.21.15.145:0/2633234420 conn(0x7fa9540a0f00 msgr2=0x7fa954144370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:36.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.224+0000 7fa95b388640 1 -- 172.21.15.145:0/2633234420 shutdown_connections 2024-04-18T10:05:36.224 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.224+0000 7fa95b388640 1 -- 172.21.15.145:0/2633234420 wait complete. 2024-04-18T10:05:36.231 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:36.231+0000 7f225fe53180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-18T10:05:36.238 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 20bac992-d716-415f-accc-1acbb8874d44 6 2024-04-18T10:05:36.239 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:36.236+0000 7f67ad67b180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-18T10:05:36.285 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:36.283+0000 7f225fe53180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-18T10:05:36.293 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:36.292+0000 7f67ad67b180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-18T10:05:36.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.344+0000 7fac27a56640 1 Processor -- start 2024-04-18T10:05:36.346 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.344+0000 7fac27a56640 1 -- start start 2024-04-18T10:05:36.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac27a56640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 0x7fac201445e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac27a56640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fac20144b20 0x7fac20147f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac27a56640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac200b0d70 con 0x7fac200afca0 2024-04-18T10:05:36.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac27a56640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fac20144fd0 con 0x7fac20144b20 2024-04-18T10:05:36.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac27a56640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fac200af170 con 0x7fac200af8c0 2024-04-18T10:05:36.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac257f2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 0x7fac201445e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac25ff3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fac20144b20 0x7fac20147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac257f2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 0x7fac201445e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:34670/0 (socket says 172.21.15.145:34670) 2024-04-18T10:05:36.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac257f2640 1 -- 172.21.15.145:0/1052004092 learned_addr learned my addr 172.21.15.145:0/1052004092 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-04-18T10:05:36.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac257f2640 1 -- 172.21.15.145:0/1052004092 >> v2:172.21.15.145:3300/0 conn(0x7fac20144b20 msgr2=0x7fac20147f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:36.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac257f2640 1 --2- 172.21.15.145:0/1052004092 >> v2:172.21.15.145:3300/0 conn(0x7fac20144b20 0x7fac20147f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac257f2640 1 -- 172.21.15.145:0/1052004092 >> v1:172.21.15.62:6789/0 conn(0x7fac200afca0 legacy=0x7fac20147840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac257f2640 1 -- 172.21.15.145:0/1052004092 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2006efc0 con 0x7fac200af8c0 2024-04-18T10:05:36.348 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac25ff3640 1 --2- 172.21.15.145:0/1052004092 >> v2:172.21.15.145:3300/0 conn(0x7fac20144b20 0x7fac20147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:05:36.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac257f2640 1 --2- 172.21.15.145:0/1052004092 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 0x7fac201445e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fac20145150 tx=0x7fac1c01c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d509b85b8e3ed0d server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:36.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac17fff640 1 -- 172.21.15.145:0/1052004092 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c0b0070 con 0x7fac200af8c0 2024-04-18T10:05:36.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac17fff640 1 -- 172.21.15.145:0/1052004092 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac1c0ab600 con 0x7fac200af8c0 2024-04-18T10:05:36.349 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac17fff640 1 -- 172.21.15.145:0/1052004092 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac1c0ab9d0 con 0x7fac200af8c0 2024-04-18T10:05:36.350 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac27a56640 1 -- 172.21.15.145:0/1052004092 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 msgr2=0x7fac201445e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.350 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac27a56640 1 --2- 172.21.15.145:0/1052004092 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 0x7fac201445e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fac20145150 tx=0x7fac1c01c4b0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.350 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac27a56640 1 -- 172.21.15.145:0/1052004092 shutdown_connections 2024-04-18T10:05:36.351 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac27a56640 1 --2- 172.21.15.145:0/1052004092 >> v2:172.21.15.145:3300/0 conn(0x7fac20144b20 0x7fac20147f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.351 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac27a56640 1 --2- 172.21.15.145:0/1052004092 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 0x7fac201445e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.351 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac27a56640 1 -- 172.21.15.145:0/1052004092 >> 172.21.15.145:0/1052004092 conn(0x7fac2006bad0 msgr2=0x7fac2006beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:36.351 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.348+0000 7fac27a56640 1 -- 172.21.15.145:0/1052004092 shutdown_connections 2024-04-18T10:05:36.351 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac27a56640 1 -- 172.21.15.145:0/1052004092 wait complete. 2024-04-18T10:05:36.351 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac27a56640 1 Processor -- start 2024-04-18T10:05:36.352 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac27a56640 1 -- start start 2024-04-18T10:05:36.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac27a56640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 0x7fac2014fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac27a56640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac200afca0 0x7fac201dddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac27a56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac20144b20 0x7fac2009e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac27a56640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fac20148610 con 0x7fac20144b20 2024-04-18T10:05:36.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac27a56640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fac20148490 con 0x7fac200afca0 2024-04-18T10:05:36.354 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac27a56640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fac2014eb70 con 0x7fac200af8c0 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac257f2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 0x7fac2014fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac24ff1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac200afca0 0x7fac201dddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac257f2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 0x7fac2014fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:34678/0 (socket says 172.21.15.145:34678) 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac24ff1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac200afca0 0x7fac201dddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:39242/0 (socket says 172.21.15.145:39242) 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac257f2640 1 -- 172.21.15.145:0/660735347 learned_addr learned my addr 172.21.15.145:0/660735347 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac25ff3640 1 --2- 172.21.15.145:0/660735347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac20144b20 0x7fac2009e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac24ff1640 1 -- 172.21.15.145:0/660735347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 msgr2=0x7fac2014fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac24ff1640 1 --2- 172.21.15.145:0/660735347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 0x7fac2014fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac24ff1640 1 -- 172.21.15.145:0/660735347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac20144b20 msgr2=0x7fac2009e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac24ff1640 1 --2- 172.21.15.145:0/660735347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac20144b20 0x7fac2009e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac24ff1640 1 -- 172.21.15.145:0/660735347 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac2009f980 con 0x7fac200afca0 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac24ff1640 1 --2- 172.21.15.145:0/660735347 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac200afca0 0x7fac201dddb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fac10000f70 tx=0x7fac1000c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac15ffb640 1 -- 172.21.15.145:0/660735347 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10017070 con 0x7fac200afca0 2024-04-18T10:05:36.355 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac15ffb640 1 -- 172.21.15.145:0/660735347 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac100052d0 con 0x7fac200afca0 2024-04-18T10:05:36.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac15ffb640 1 -- 172.21.15.145:0/660735347 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac100054c0 con 0x7fac200afca0 2024-04-18T10:05:36.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac27a56640 1 -- 172.21.15.145:0/660735347 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac2009d9a0 con 0x7fac200afca0 2024-04-18T10:05:36.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac27a56640 1 -- 172.21.15.145:0/660735347 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac2009efc0 con 0x7fac200afca0 2024-04-18T10:05:36.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac15ffb640 1 -- 172.21.15.145:0/660735347 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fac1001a070 con 0x7fac200afca0 2024-04-18T10:05:36.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.352+0000 7fac15ffb640 1 -- 172.21.15.145:0/660735347 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fac10011b30 con 0x7fac200afca0 2024-04-18T10:05:36.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.356+0000 7fac27a56640 1 -- 172.21.15.145:0/660735347 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac20069230 con 0x7fac200afca0 2024-04-18T10:05:36.356 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.356+0000 7fac15ffb640 1 -- 172.21.15.145:0/660735347 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fac10021020 con 0x7fac200afca0 2024-04-18T10:05:36.367 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:36.367+0000 7f225fe53180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-18T10:05:36.422 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:36.423+0000 7f225fe53180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-18T10:05:36.423 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.424+0000 7fac27a56640 1 -- 172.21.15.145:0/660735347 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "20bac992-d716-415f-accc-1acbb8874d44", "id": 6} v 0) -- 0x7fac200b0a20 con 0x7fac200afca0 2024-04-18T10:05:36.430 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.432+0000 7fac15ffb640 1 -- 172.21.15.145:0/660735347 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "20bac992-d716-415f-accc-1acbb8874d44", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fac10011e10 con 0x7fac200afca0 2024-04-18T10:05:36.430 INFO:teuthology.orchestra.run.smithi145.stdout:6 2024-04-18T10:05:36.432 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.432+0000 7fac27a56640 1 -- 172.21.15.145:0/660735347 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac200afca0 msgr2=0x7fac201dddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.432 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.432+0000 7fac27a56640 1 --2- 172.21.15.145:0/660735347 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac200afca0 0x7fac201dddb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fac10000f70 tx=0x7fac1000c890 comp rx=0 tx=0).stop 2024-04-18T10:05:36.432 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.432+0000 7fac27a56640 1 -- 172.21.15.145:0/660735347 shutdown_connections 2024-04-18T10:05:36.432 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.432+0000 7fac27a56640 1 --2- 172.21.15.145:0/660735347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac20144b20 0x7fac2009e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.432 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.432+0000 7fac27a56640 1 --2- 172.21.15.145:0/660735347 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac200afca0 0x7fac201dddb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.432 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.432+0000 7fac27a56640 1 --2- 172.21.15.145:0/660735347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac200af8c0 0x7fac2014fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.432 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.432+0000 7fac27a56640 1 -- 172.21.15.145:0/660735347 >> 172.21.15.145:0/660735347 conn(0x7fac2006bad0 msgr2=0x7fac20074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:36.432 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.432+0000 7fac27a56640 1 -- 172.21.15.145:0/660735347 shutdown_connections 2024-04-18T10:05:36.432 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.432+0000 7fac27a56640 1 -- 172.21.15.145:0/660735347 wait complete. 2024-04-18T10:05:36.446 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 7de29c9e-17ce-496a-bada-ec9b482f8f04 7 2024-04-18T10:05:36.452 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:36.452+0000 7f67ad67b180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-18T10:05:36.515 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:36.516+0000 7f67ad67b180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-18T10:05:36.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f7b311640 1 Processor -- start 2024-04-18T10:05:36.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f7b311640 1 -- start start 2024-04-18T10:05:36.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f7b311640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f74067a70 0x7f6f7413f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f7b311640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6f7406ef10 0x7f6f7413fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f7b311640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f74067e50 con 0x7f6f740b6310 2024-04-18T10:05:36.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f7b311640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f6f74070ed0 con 0x7f6f7406ef10 2024-04-18T10:05:36.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f7b311640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f6f74071120 con 0x7f6f74067a70 2024-04-18T10:05:36.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f790ad640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f74067a70 0x7f6f7413f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f798ae640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f6f740b6310 0x7f6f74072d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:60048/0 (socket says 172.21.15.145:60048) 2024-04-18T10:05:36.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f788ac640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6f7406ef10 0x7f6f7413fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f790ad640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f74067a70 0x7f6f7413f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:34680/0 (socket says 172.21.15.145:34680) 2024-04-18T10:05:36.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f798ae640 1 -- 172.21.15.145:0/2608457603 learned_addr learned my addr 172.21.15.145:0/2608457603 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-04-18T10:05:36.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f790ad640 1 -- 172.21.15.145:0/2608457603 >> v2:172.21.15.145:3300/0 conn(0x7f6f7406ef10 msgr2=0x7f6f7413fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:36.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f790ad640 1 --2- 172.21.15.145:0/2608457603 >> v2:172.21.15.145:3300/0 conn(0x7f6f7406ef10 0x7f6f7413fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f790ad640 1 -- 172.21.15.145:0/2608457603 >> v1:172.21.15.145:6789/0 conn(0x7f6f740b6310 legacy=0x7f6f74072d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f790ad640 1 -- 172.21.15.145:0/2608457603 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f74071400 con 0x7f6f74067a70 2024-04-18T10:05:36.557 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f788ac640 1 --2- 172.21.15.145:0/2608457603 >> v2:172.21.15.145:3300/0 conn(0x7f6f7406ef10 0x7f6f7413fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:05:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.556+0000 7f6f790ad640 1 --2- 172.21.15.145:0/2608457603 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f74067a70 0x7f6f7413f890 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6f700097c0 tx=0x7f6f7001c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d95dc29c4fb3c631 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:36.558 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f6bfff640 1 -- 172.21.15.145:0/2608457603 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f700b0070 con 0x7f6f74067a70 2024-04-18T10:05:36.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f6bfff640 1 -- 172.21.15.145:0/2608457603 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f700ab600 con 0x7f6f74067a70 2024-04-18T10:05:36.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f6bfff640 1 -- 172.21.15.145:0/2608457603 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f700ab9d0 con 0x7f6f74067a70 2024-04-18T10:05:36.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 -- 172.21.15.145:0/2608457603 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f74067a70 msgr2=0x7f6f7413f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.559 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 --2- 172.21.15.145:0/2608457603 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f74067a70 0x7f6f7413f890 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6f700097c0 tx=0x7f6f7001c380 comp rx=0 tx=0).stop 2024-04-18T10:05:36.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 -- 172.21.15.145:0/2608457603 shutdown_connections 2024-04-18T10:05:36.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 --2- 172.21.15.145:0/2608457603 >> v2:172.21.15.145:3300/0 conn(0x7f6f7406ef10 0x7f6f7413fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 --2- 172.21.15.145:0/2608457603 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f74067a70 0x7f6f7413f890 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 -- 172.21.15.145:0/2608457603 >> 172.21.15.145:0/2608457603 conn(0x7f6f7406bad0 msgr2=0x7f6f7406beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:36.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 -- 172.21.15.145:0/2608457603 shutdown_connections 2024-04-18T10:05:36.560 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 -- 172.21.15.145:0/2608457603 wait complete. 2024-04-18T10:05:36.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 Processor -- start 2024-04-18T10:05:36.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 -- start start 2024-04-18T10:05:36.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f74067a70 0x7f6f74148280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f7406ef10 0x7f6f740b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f740b6310 0x7f6f740b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6f74152fe0 con 0x7f6f74067a70 2024-04-18T10:05:36.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f6f74152e60 con 0x7f6f740b6310 2024-04-18T10:05:36.562 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f6f74153160 con 0x7f6f7406ef10 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f788ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f7406ef10 0x7f6f740b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f788ac640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f7406ef10 0x7f6f740b34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:34696/0 (socket says 172.21.15.145:34696) 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f788ac640 1 -- 172.21.15.145:0/28490970 learned_addr learned my addr 172.21.15.145:0/28490970 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f798ae640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f740b6310 0x7f6f740b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f790ad640 1 --2- 172.21.15.145:0/28490970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f74067a70 0x7f6f74148280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f788ac640 1 -- 172.21.15.145:0/28490970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f740b6310 msgr2=0x7f6f740b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f788ac640 1 --2- 172.21.15.145:0/28490970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f740b6310 0x7f6f740b3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f788ac640 1 -- 172.21.15.145:0/28490970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f74067a70 msgr2=0x7f6f74148280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f788ac640 1 --2- 172.21.15.145:0/28490970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f74067a70 0x7f6f74148280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f788ac640 1 -- 172.21.15.145:0/28490970 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f74147200 con 0x7f6f7406ef10 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f798ae640 1 --2- 172.21.15.145:0/28490970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f740b6310 0x7f6f740b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f790ad640 1 --2- 172.21.15.145:0/28490970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f74067a70 0x7f6f74148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f788ac640 1 --2- 172.21.15.145:0/28490970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f7406ef10 0x7f6f740b34e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400e9d0 tx=0x7f6f6400ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:36.563 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f69ffb640 1 -- 172.21.15.145:0/28490970 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f64017070 con 0x7f6f7406ef10 2024-04-18T10:05:36.564 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f69ffb640 1 -- 172.21.15.145:0/28490970 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f6400ca60 con 0x7f6f7406ef10 2024-04-18T10:05:36.564 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f69ffb640 1 -- 172.21.15.145:0/28490970 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6400ce50 con 0x7f6f7406ef10 2024-04-18T10:05:36.564 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 -- 172.21.15.145:0/28490970 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f74148d40 con 0x7f6f7406ef10 2024-04-18T10:05:36.564 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.560+0000 7f6f7b311640 1 -- 172.21.15.145:0/28490970 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f741478f0 con 0x7f6f7406ef10 2024-04-18T10:05:36.564 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.564+0000 7f6f69ffb640 1 -- 172.21.15.145:0/28490970 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6f6401a070 con 0x7f6f7406ef10 2024-04-18T10:05:36.564 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.564+0000 7f6f69ffb640 1 -- 172.21.15.145:0/28490970 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f6f64003c80 con 0x7f6f7406ef10 2024-04-18T10:05:36.564 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.564+0000 7f6f7b311640 1 -- 172.21.15.145:0/28490970 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f74069230 con 0x7f6f7406ef10 2024-04-18T10:05:36.565 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.564+0000 7f6f69ffb640 1 -- 172.21.15.145:0/28490970 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f6f640059a0 con 0x7f6f7406ef10 2024-04-18T10:05:36.580 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:36.579+0000 7f225fe53180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-18T10:05:36.629 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.628+0000 7f6f7b311640 1 -- 172.21.15.145:0/28490970 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7de29c9e-17ce-496a-bada-ec9b482f8f04", "id": 7} v 0) -- 0x7f6f74143e30 con 0x7f6f7406ef10 2024-04-18T10:05:36.639 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.640+0000 7f6f69ffb640 1 -- 172.21.15.145:0/28490970 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7de29c9e-17ce-496a-bada-ec9b482f8f04", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f6f64003620 con 0x7f6f7406ef10 2024-04-18T10:05:36.639 INFO:teuthology.orchestra.run.smithi145.stdout:7 2024-04-18T10:05:36.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.640+0000 7f6f7b311640 1 -- 172.21.15.145:0/28490970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f7406ef10 msgr2=0x7f6f740b34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.640+0000 7f6f7b311640 1 --2- 172.21.15.145:0/28490970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f7406ef10 0x7f6f740b34e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6f6400e9d0 tx=0x7f6f6400ee90 comp rx=0 tx=0).stop 2024-04-18T10:05:36.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.640+0000 7f6f7b311640 1 -- 172.21.15.145:0/28490970 shutdown_connections 2024-04-18T10:05:36.640 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.640+0000 7f6f7b311640 1 --2- 172.21.15.145:0/28490970 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6f740b6310 0x7f6f740b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.640+0000 7f6f7b311640 1 --2- 172.21.15.145:0/28490970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6f7406ef10 0x7f6f740b34e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.640+0000 7f6f7b311640 1 --2- 172.21.15.145:0/28490970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f74067a70 0x7f6f74148280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.640+0000 7f6f7b311640 1 -- 172.21.15.145:0/28490970 >> 172.21.15.145:0/28490970 conn(0x7f6f7406bad0 msgr2=0x7f6f74136a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:36.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.640+0000 7f6f7b311640 1 -- 172.21.15.145:0/28490970 shutdown_connections 2024-04-18T10:05:36.641 INFO:teuthology.orchestra.run.smithi145.stderr:2024-04-18T10:05:36.640+0000 7f6f7b311640 1 -- 172.21.15.145:0/28490970 wait complete. 2024-04-18T10:05:36.644 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:36.643+0000 7f225fe53180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-18T10:05:36.655 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-18T10:05:36.655 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-18T10:05:36.657 INFO:tasks.ceph.osd.0:Started 2024-04-18T10:05:36.657 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-18T10:05:36.657 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-18T10:05:36.659 INFO:tasks.ceph.osd.1:Started 2024-04-18T10:05:36.659 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-18T10:05:36.660 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-18T10:05:36.661 INFO:tasks.ceph.osd.2:Started 2024-04-18T10:05:36.661 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-18T10:05:36.662 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-18T10:05:36.663 INFO:tasks.ceph.osd.3:Started 2024-04-18T10:05:36.664 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-18T10:05:36.664 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-18T10:05:36.699 INFO:tasks.ceph.osd.4:Started 2024-04-18T10:05:36.699 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-18T10:05:36.700 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-18T10:05:36.701 INFO:tasks.ceph.osd.5:Started 2024-04-18T10:05:36.702 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-18T10:05:36.702 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-18T10:05:36.703 INFO:tasks.ceph.osd.6:Started 2024-04-18T10:05:36.703 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-18T10:05:36.704 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-18T10:05:36.705 INFO:tasks.ceph.osd.7:Started 2024-04-18T10:05:36.706 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:05:36.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ecaa640 1 Processor -- start 2024-04-18T10:05:36.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ecaa640 1 -- start start 2024-04-18T10:05:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ecaa640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1348067a70 0x7f13481418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ecaa640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f13481482d0 0x7f1348152540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ecaa640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1348067e50 con 0x7f1348141e00 2024-04-18T10:05:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ecaa640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f1348070c30 con 0x7f1348067a70 2024-04-18T10:05:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ecaa640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f1348070e80 con 0x7f13481482d0 2024-04-18T10:05:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ca46640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1348067a70 0x7f13481418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ca46640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1348067a70 0x7f13481418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54376/0 (socket says 172.21.15.62:54376) 2024-04-18T10:05:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ca46640 1 -- 172.21.15.62:0/3627841579 learned_addr learned my addr 172.21.15.62:0/3627841579 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134d247640 1 --2- 172.21.15.62:0/3627841579 >> v2:172.21.15.145:3300/0 conn(0x7f13481482d0 0x7f1348152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ca46640 1 -- 172.21.15.62:0/3627841579 >> v2:172.21.15.145:3300/0 conn(0x7f13481482d0 msgr2=0x7f1348152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ca46640 1 --2- 172.21.15.62:0/3627841579 >> v2:172.21.15.145:3300/0 conn(0x7f13481482d0 0x7f1348152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ca46640 1 -- 172.21.15.62:0/3627841579 >> v1:172.21.15.145:6789/0 conn(0x7f1348141e00 legacy=0x7f1348147bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.823+0000 7f134ca46640 1 -- 172.21.15.62:0/3627841579 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1348072ad0 con 0x7f1348067a70 2024-04-18T10:05:36.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f134ca46640 1 --2- 172.21.15.62:0/3627841579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1348067a70 0x7f13481418c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1348073f50 tx=0x7f133c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d42db700ba997d server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:36.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f13477fe640 1 -- 172.21.15.62:0/3627841579 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0ab430 con 0x7f1348067a70 2024-04-18T10:05:36.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f13477fe640 1 -- 172.21.15.62:0/3627841579 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f133c0b0070 con 0x7f1348067a70 2024-04-18T10:05:36.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f13477fe640 1 -- 172.21.15.62:0/3627841579 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0ab7f0 con 0x7f1348067a70 2024-04-18T10:05:36.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f134ecaa640 1 -- 172.21.15.62:0/3627841579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1348067a70 msgr2=0x7f13481418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f134ecaa640 1 --2- 172.21.15.62:0/3627841579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1348067a70 0x7f13481418c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1348073f50 tx=0x7f133c01c520 comp rx=0 tx=0).stop 2024-04-18T10:05:36.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f134ecaa640 1 -- 172.21.15.62:0/3627841579 shutdown_connections 2024-04-18T10:05:36.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f134ecaa640 1 --2- 172.21.15.62:0/3627841579 >> v2:172.21.15.145:3300/0 conn(0x7f13481482d0 0x7f1348152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f134ecaa640 1 --2- 172.21.15.62:0/3627841579 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1348067a70 0x7f13481418c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f134ecaa640 1 -- 172.21.15.62:0/3627841579 >> 172.21.15.62:0/3627841579 conn(0x7f134806bad0 msgr2=0x7f134806bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:36.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f134ecaa640 1 -- 172.21.15.62:0/3627841579 shutdown_connections 2024-04-18T10:05:36.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f134ecaa640 1 -- 172.21.15.62:0/3627841579 wait complete. 2024-04-18T10:05:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f134ecaa640 1 Processor -- start 2024-04-18T10:05:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.827+0000 7f134ecaa640 1 -- start start 2024-04-18T10:05:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ecaa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1348067a70 0x7f13480b9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ecaa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1348141e00 0x7f13481e0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ecaa640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13481482d0 0x7f13481e0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ecaa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f13480719e0 con 0x7f1348067a70 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ecaa640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f1348071860 con 0x7f13481482d0 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ecaa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f13481552a0 con 0x7f1348141e00 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ca46640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1348067a70 0x7f13480b9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ca46640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1348067a70 0x7f13480b9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54382/0 (socket says 172.21.15.62:54382) 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ca46640 1 -- 172.21.15.62:0/274857531 learned_addr learned my addr 172.21.15.62:0/274857531 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f1347fff640 1 --2- 172.21.15.62:0/274857531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1348141e00 0x7f13481e0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134d247640 1 --2- 172.21.15.62:0/274857531 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13481482d0 0x7f13481e0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ca46640 1 -- 172.21.15.62:0/274857531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1348141e00 msgr2=0x7f13481e0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ca46640 1 --2- 172.21.15.62:0/274857531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1348141e00 0x7f13481e0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ca46640 1 -- 172.21.15.62:0/274857531 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13481482d0 msgr2=0x7f13481e0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ca46640 1 --2- 172.21.15.62:0/274857531 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13481482d0 0x7f13481e0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ca46640 1 -- 172.21.15.62:0/274857531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13480b87f0 con 0x7f1348067a70 2024-04-18T10:05:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f1347fff640 1 --2- 172.21.15.62:0/274857531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1348141e00 0x7f13481e0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:05:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ca46640 1 --2- 172.21.15.62:0/274857531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1348067a70 0x7f13480b9b70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f133c01abd0 tx=0x7f133c01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f13457fa640 1 -- 172.21.15.62:0/274857531 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0be5f0 con 0x7f1348067a70 2024-04-18T10:05:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f13457fa640 1 -- 172.21.15.62:0/274857531 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f133c0be790 con 0x7f1348067a70 2024-04-18T10:05:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ecaa640 1 -- 172.21.15.62:0/274857531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13480b8ad0 con 0x7f1348067a70 2024-04-18T10:05:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f13457fa640 1 -- 172.21.15.62:0/274857531 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c0beb60 con 0x7f1348067a70 2024-04-18T10:05:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ecaa640 1 -- 172.21.15.62:0/274857531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13480ba0b0 con 0x7f1348067a70 2024-04-18T10:05:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f13457fa640 1 -- 172.21.15.62:0/274857531 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f133c0b4780 con 0x7f1348067a70 2024-04-18T10:05:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f13457fa640 1 -- 172.21.15.62:0/274857531 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f133c0a8070 con 0x7f1348067a70 2024-04-18T10:05:36.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.831+0000 7f134ecaa640 1 -- 172.21.15.62:0/274857531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1348069230 con 0x7f1348067a70 2024-04-18T10:05:36.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.843+0000 7f13457fa640 1 -- 172.21.15.62:0/274857531 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f133c016110 con 0x7f1348067a70 2024-04-18T10:05:36.900 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.899+0000 7f134ecaa640 1 -- 172.21.15.62:0/274857531 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13480b5570 con 0x7f1348067a70 2024-04-18T10:05:36.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.899+0000 7f13457fa640 1 -- 172.21.15.62:0/274857531 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f133c0b3380 con 0x7f1348067a70 2024-04-18T10:05:36.901 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:36.901 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":10,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:05:36.635206+0000","last_up_change":"0.000000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:05:36.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.899+0000 7f134ecaa640 1 -- 172.21.15.62:0/274857531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1348067a70 msgr2=0x7f13480b9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:36.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.899+0000 7f134ecaa640 1 --2- 172.21.15.62:0/274857531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1348067a70 0x7f13480b9b70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f133c01abd0 tx=0x7f133c01ad60 comp rx=0 tx=0).stop 2024-04-18T10:05:36.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.899+0000 7f134ecaa640 1 -- 172.21.15.62:0/274857531 shutdown_connections 2024-04-18T10:05:36.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.899+0000 7f134ecaa640 1 --2- 172.21.15.62:0/274857531 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13481482d0 0x7f13481e0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.899+0000 7f134ecaa640 1 --2- 172.21.15.62:0/274857531 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1348141e00 0x7f13481e0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.899+0000 7f134ecaa640 1 --2- 172.21.15.62:0/274857531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1348067a70 0x7f13480b9b70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:36.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.899+0000 7f134ecaa640 1 -- 172.21.15.62:0/274857531 >> 172.21.15.62:0/274857531 conn(0x7f134806bad0 msgr2=0x7f134813d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:36.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.899+0000 7f134ecaa640 1 -- 172.21.15.62:0/274857531 shutdown_connections 2024-04-18T10:05:36.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:36.899+0000 7f134ecaa640 1 -- 172.21.15.62:0/274857531 wait complete. 2024-04-18T10:05:36.916 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-18T10:05:36.917 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-18T10:05:37.134 INFO:tasks.ceph.mgr.x.smithi145.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-18T10:05:37.135 INFO:tasks.ceph.mgr.x.smithi145.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-18T10:05:37.135 INFO:tasks.ceph.mgr.x.smithi145.stderr: from numpy import show_config as show_numpy_config 2024-04-18T10:05:37.212 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:37.212+0000 7f67ad67b180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-18T10:05:37.288 INFO:tasks.ceph.osd.1.smithi062.stderr:2024-04-18T10:05:37.287+0000 7fc68c5df7c0 -1 Falling back to public interface 2024-04-18T10:05:37.288 INFO:tasks.ceph.osd.3.smithi062.stderr:2024-04-18T10:05:37.287+0000 7fbdc86e97c0 -1 Falling back to public interface 2024-04-18T10:05:37.300 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-04-18T10:05:37.299+0000 7f3f2da467c0 -1 Falling back to public interface 2024-04-18T10:05:37.300 INFO:tasks.ceph.osd.2.smithi062.stderr:2024-04-18T10:05:37.299+0000 7f858c9c77c0 -1 Falling back to public interface 2024-04-18T10:05:37.316 INFO:tasks.ceph.osd.6.smithi145.stderr:2024-04-18T10:05:37.316+0000 7fab6ecfd7c0 -1 Falling back to public interface 2024-04-18T10:05:37.316 INFO:tasks.ceph.osd.7.smithi145.stderr:2024-04-18T10:05:37.316+0000 7f1509e587c0 -1 Falling back to public interface 2024-04-18T10:05:37.320 INFO:tasks.ceph.osd.4.smithi145.stderr:2024-04-18T10:05:37.320+0000 7f4988e5e7c0 -1 Falling back to public interface 2024-04-18T10:05:37.327 INFO:tasks.ceph.osd.5.smithi145.stderr:2024-04-18T10:05:37.328+0000 7fa5f6ff77c0 -1 Falling back to public interface 2024-04-18T10:05:37.401 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:37.400+0000 7f67ad67b180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-18T10:05:37.450 INFO:tasks.ceph.mgr.y.smithi062.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-18T10:05:37.450 INFO:tasks.ceph.mgr.y.smithi062.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-18T10:05:37.450 INFO:tasks.ceph.mgr.y.smithi062.stderr: from numpy import show_config as show_numpy_config 2024-04-18T10:05:37.490 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:37.487+0000 7f225fe53180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-18T10:05:37.579 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:37.580+0000 7f67ad67b180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-18T10:05:37.632 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:37.631+0000 7f225fe53180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-18T10:05:37.649 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:37.648+0000 7f67ad67b180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-18T10:05:37.722 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-18T10:05:37.796 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:37.796+0000 7f67ad67b180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-18T10:05:37.835 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db7d22640 1 Processor -- start 2024-04-18T10:05:37.835 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db7d22640 1 -- start start 2024-04-18T10:05:37.838 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db7d22640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8db006cd40 0x7f8db0072e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:37.838 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db7d22640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8db0140020 0x7f8db0150430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:37.838 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db7d22640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8db00af8a0 con 0x7f8db00b5e20 2024-04-18T10:05:37.838 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db7d22640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f8db00b0700 con 0x7f8db006cd40 2024-04-18T10:05:37.838 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db7d22640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8db00afb10 con 0x7f8db0140020 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db52bd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8db00b5e20 0x7f8db013f910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:50838/0 (socket says 172.21.15.62:50838) 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db52bd640 1 -- 172.21.15.62:0/1688887940 learned_addr learned my addr 172.21.15.62:0/1688887940 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db62bf640 1 --2- 172.21.15.62:0/1688887940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8db0140020 0x7f8db0150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db5abe640 1 --2- 172.21.15.62:0/1688887940 >> v2:172.21.15.145:3300/0 conn(0x7f8db006cd40 0x7f8db0072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 372421436 0 0) 0x7f8db00af8a0 con 0x7f8db00b5e20 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d9c003530 con 0x7f8db00b5e20 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1199881124 0 0) 0x7f8d9c003530 con 0x7f8db00b5e20 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8db0140020 msgr2=0x7f8db0150430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 --2- 172.21.15.62:0/1688887940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8db0140020 0x7f8db0150430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 >> v2:172.21.15.145:3300/0 conn(0x7f8db006cd40 msgr2=0x7f8db0072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 --2- 172.21.15.62:0/1688887940 >> v2:172.21.15.145:3300/0 conn(0x7f8db006cd40 0x7f8db0072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db006d120 con 0x7f8db00b5e20 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f8da0001680 con 0x7f8db00b5e20 2024-04-18T10:05:37.839 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 >> v1:172.21.15.62:6789/0 conn(0x7f8db00b5e20 legacy=0x7f8db013f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 --2- 172.21.15.62:0/1688887940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d9c0039c0 0x7f8d9c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 --2- 172.21.15.62:0/1688887940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8db0140020 0x7f8d9c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 --2- 172.21.15.62:0/1688887940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d9c004d50 0x7f8d9c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8db00b0700 con 0x7f8d9c0039c0 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f8db00afb10 con 0x7f8d9c004d50 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8d9c00b9d0 con 0x7f8db0140020 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db5abe640 1 --2- 172.21.15.62:0/1688887940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d9c0039c0 0x7f8d9c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db52bd640 1 --2- 172.21.15.62:0/1688887940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8db0140020 0x7f8d9c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db62bf640 1 --2- 172.21.15.62:0/1688887940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d9c004d50 0x7f8d9c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db5abe640 1 -- 172.21.15.62:0/1688887940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8db0140020 msgr2=0x7f8d9c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db5abe640 1 --2- 172.21.15.62:0/1688887940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8db0140020 0x7f8d9c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db5abe640 1 -- 172.21.15.62:0/1688887940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d9c004d50 msgr2=0x7f8d9c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db5abe640 1 --2- 172.21.15.62:0/1688887940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8d9c004d50 0x7f8d9c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:37.840 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.835+0000 7f8db5abe640 1 -- 172.21.15.62:0/1688887940 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9c003530 con 0x7f8d9c0039c0 2024-04-18T10:05:37.843 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db5abe640 1 --2- 172.21.15.62:0/1688887940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d9c0039c0 0x7f8d9c003de0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009b10 tx=0x7f8dac01cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de31dd05be8b014 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac0b2070 con 0x7f8d9c0039c0 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dac0a5680 con 0x7f8d9c0039c0 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db4abc640 1 -- 172.21.15.62:0/1688887940 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac0096b0 con 0x7f8d9c0039c0 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 -- 172.21.15.62:0/1688887940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d9c0039c0 msgr2=0x7f8d9c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 --2- 172.21.15.62:0/1688887940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d9c0039c0 0x7f8d9c003de0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8dac009b10 tx=0x7f8dac01cab0 comp rx=0 tx=0).stop 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db62bf640 1 -- 172.21.15.62:0/1688887940 reap_dead start 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 -- 172.21.15.62:0/1688887940 shutdown_connections 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 -- 172.21.15.62:0/1688887940 >> 172.21.15.62:0/1688887940 conn(0x7f8db00b6b70 msgr2=0x7f8db00b9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 -- 172.21.15.62:0/1688887940 shutdown_connections 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 -- 172.21.15.62:0/1688887940 wait complete. 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 Processor -- start 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 -- start start 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d9c0039c0 0x7f8db01ddad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db0143e00 0x7f8db0144990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:37.844 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8db0142690 0x7f8db0142b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8db00b0b70 con 0x7f8db0143e00 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f8db00b0390 con 0x7f8db0142690 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db7d22640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8db00b0cf0 con 0x7f8d9c0039c0 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db52bd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db0143e00 0x7f8db0144990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db52bd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db0143e00 0x7f8db0144990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:54408/0 (socket says 172.21.15.62:54408) 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db52bd640 1 -- 172.21.15.62:0/3695660691 learned_addr learned my addr 172.21.15.62:0/3695660691 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db62bf640 1 --2- 172.21.15.62:0/3695660691 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8db0142690 0x7f8db0142b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.839+0000 7f8db5abe640 1 --2- 172.21.15.62:0/3695660691 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d9c0039c0 0x7f8db01ddad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8db52bd640 1 -- 172.21.15.62:0/3695660691 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d9c0039c0 msgr2=0x7f8db01ddad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8db52bd640 1 --2- 172.21.15.62:0/3695660691 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d9c0039c0 0x7f8db01ddad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8db52bd640 1 -- 172.21.15.62:0/3695660691 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8db0142690 msgr2=0x7f8db0142b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8db52bd640 1 --2- 172.21.15.62:0/3695660691 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8db0142690 0x7f8db0142b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8db52bd640 1 -- 172.21.15.62:0/3695660691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db0145ef0 con 0x7f8db0143e00 2024-04-18T10:05:37.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8db52bd640 1 --2- 172.21.15.62:0/3695660691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db0143e00 0x7f8db0144990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8da0001870 tx=0x7f8da000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:37.846 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8da67fc640 1 -- 172.21.15.62:0/3695660691 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da000c8d0 con 0x7f8db0143e00 2024-04-18T10:05:37.846 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8da67fc640 1 -- 172.21.15.62:0/3695660691 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da0018070 con 0x7f8db0143e00 2024-04-18T10:05:37.846 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8da67fc640 1 -- 172.21.15.62:0/3695660691 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da000ccb0 con 0x7f8db0143e00 2024-04-18T10:05:37.846 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8db7d22640 1 -- 172.21.15.62:0/3695660691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db0144260 con 0x7f8db0143e00 2024-04-18T10:05:37.846 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8db7d22640 1 -- 172.21.15.62:0/3695660691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db00b3b80 con 0x7f8db0143e00 2024-04-18T10:05:37.846 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8da67fc640 1 -- 172.21.15.62:0/3695660691 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8da001d460 con 0x7f8db0143e00 2024-04-18T10:05:37.846 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8da67fc640 1 -- 172.21.15.62:0/3695660691 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f8da0013dc0 con 0x7f8db0143e00 2024-04-18T10:05:37.846 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8db7d22640 1 -- 172.21.15.62:0/3695660691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8db0143a20 con 0x7f8db0143e00 2024-04-18T10:05:37.847 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.843+0000 7f8da67fc640 1 -- 172.21.15.62:0/3695660691 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8db0144260 con 0x7f8db0143e00 2024-04-18T10:05:37.895 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:37.895+0000 7f225fe53180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-18T10:05:37.903 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.903+0000 7f8db7d22640 1 -- 172.21.15.62:0/3695660691 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8db00af550 con 0x7f8db0143e00 2024-04-18T10:05:37.904 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.903+0000 7f8da67fc640 1 -- 172.21.15.62:0/3695660691 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f8da00136f0 con 0x7f8db0143e00 2024-04-18T10:05:37.904 INFO:teuthology.misc.health.smithi062.stdout: 2024-04-18T10:05:37.904 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":10,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:05:36.635206+0000","last_up_change":"0.000000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:05:37.905 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.903+0000 7f8db7d22640 1 -- 172.21.15.62:0/3695660691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db0143e00 msgr2=0x7f8db0144990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:37.905 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.903+0000 7f8db7d22640 1 --2- 172.21.15.62:0/3695660691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db0143e00 0x7f8db0144990 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8da0001870 tx=0x7f8da000ece0 comp rx=0 tx=0).stop 2024-04-18T10:05:37.905 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.903+0000 7f8db7d22640 1 -- 172.21.15.62:0/3695660691 shutdown_connections 2024-04-18T10:05:37.905 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.903+0000 7f8db7d22640 1 --2- 172.21.15.62:0/3695660691 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8db0142690 0x7f8db0142b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:37.905 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.903+0000 7f8db7d22640 1 --2- 172.21.15.62:0/3695660691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8db0143e00 0x7f8db0144990 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:37.905 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.903+0000 7f8db7d22640 1 --2- 172.21.15.62:0/3695660691 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8d9c0039c0 0x7f8db01ddad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:37.905 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.903+0000 7f8db7d22640 1 -- 172.21.15.62:0/3695660691 >> 172.21.15.62:0/3695660691 conn(0x7f8db00b6b70 msgr2=0x7f8db013a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:37.905 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.903+0000 7f8db7d22640 1 -- 172.21.15.62:0/3695660691 shutdown_connections 2024-04-18T10:05:37.905 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:37.903+0000 7f8db7d22640 1 -- 172.21.15.62:0/3695660691 wait complete. 2024-04-18T10:05:37.919 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-18T10:05:37.963 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:37.963+0000 7f225fe53180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-18T10:05:38.104 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:38.103+0000 7f225fe53180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-18T10:05:38.406 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:38.404+0000 7f67ad67b180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-18T10:05:38.480 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:38.480+0000 7f67ad67b180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-18T10:05:38.661 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:38.660+0000 7f67ad67b180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-18T10:05:38.734 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:38.732+0000 7f67ad67b180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-18T10:05:38.742 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:38.739+0000 7f225fe53180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-18T10:05:38.821 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:38.819+0000 7f225fe53180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-18T10:05:39.012 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:39.011+0000 7f225fe53180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-18T10:05:39.022 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:39.020+0000 7f67ad67b180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-18T10:05:39.082 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:39.084+0000 7f67ad67b180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-18T10:05:39.089 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:39.087+0000 7f225fe53180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-18T10:05:39.401 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:39.399+0000 7f225fe53180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-18T10:05:39.466 INFO:tasks.ceph.mgr.y.smithi062.stderr:2024-04-18T10:05:39.463+0000 7f225fe53180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-18T10:05:40.122 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:40.120+0000 7f67549c1640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-18T10:05:40.122 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-04-18T10:05:40.120+0000 7f67549c1640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-18T10:05:40.963 INFO:tasks.ceph.osd.5.smithi145.stderr:2024-04-18T10:05:40.964+0000 7fa5f6ff77c0 -1 osd.5 0 log_to_monitors true 2024-04-18T10:05:41.069 INFO:tasks.ceph.osd.3.smithi062.stderr:2024-04-18T10:05:41.067+0000 7fbdc86e97c0 -1 osd.3 0 log_to_monitors true 2024-04-18T10:05:41.141 INFO:tasks.ceph.osd.6.smithi145.stderr:2024-04-18T10:05:41.140+0000 7fab6ecfd7c0 -1 osd.6 0 log_to_monitors true 2024-04-18T10:05:41.164 INFO:tasks.ceph.osd.1.smithi062.stderr:2024-04-18T10:05:41.163+0000 7fc68c5df7c0 -1 osd.1 0 log_to_monitors true 2024-04-18T10:05:41.169 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-04-18T10:05:41.167+0000 7f3f2da467c0 -1 osd.0 0 log_to_monitors true 2024-04-18T10:05:41.169 INFO:tasks.ceph.osd.7.smithi145.stderr:2024-04-18T10:05:41.168+0000 7f1509e587c0 -1 osd.7 0 log_to_monitors true 2024-04-18T10:05:41.261 INFO:tasks.ceph.osd.4.smithi145.stderr:2024-04-18T10:05:41.260+0000 7f4988e5e7c0 -1 osd.4 0 log_to_monitors true 2024-04-18T10:05:41.264 INFO:tasks.ceph.osd.2.smithi062.stderr:2024-04-18T10:05:41.263+0000 7f858c9c77c0 -1 osd.2 0 log_to_monitors true 2024-04-18T10:05:44.411 INFO:tasks.ceph.osd.6.smithi145.stderr:2024-04-18T10:05:44.412+0000 7fab6aca6640 -1 osd.6 0 waiting for initial osdmap 2024-04-18T10:05:44.450 INFO:tasks.ceph.osd.6.smithi145.stderr:2024-04-18T10:05:44.448+0000 7fab62ada640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T10:05:44.498 INFO:tasks.ceph.osd.4.smithi145.stderr:2024-04-18T10:05:44.496+0000 7f4985619640 -1 osd.4 0 waiting for initial osdmap 2024-04-18T10:05:44.531 INFO:tasks.ceph.osd.4.smithi145.stderr:2024-04-18T10:05:44.532+0000 7f497cc3b640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T10:05:44.548 INFO:tasks.ceph.osd.5.smithi145.stderr:2024-04-18T10:05:44.548+0000 7fa5f2fa0640 -1 osd.5 0 waiting for initial osdmap 2024-04-18T10:05:44.555 INFO:tasks.ceph.osd.2.smithi062.stderr:2024-04-18T10:05:44.559+0000 7f8588970640 -1 osd.2 0 waiting for initial osdmap 2024-04-18T10:05:44.577 INFO:tasks.ceph.osd.7.smithi145.stderr:2024-04-18T10:05:44.576+0000 7f1505e01640 -1 osd.7 0 waiting for initial osdmap 2024-04-18T10:05:44.579 INFO:tasks.ceph.osd.5.smithi145.stderr:2024-04-18T10:05:44.580+0000 7fa5eadd4640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T10:05:44.583 INFO:tasks.ceph.osd.2.smithi062.stderr:2024-04-18T10:05:44.587+0000 7f85807a4640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T10:05:44.598 INFO:tasks.ceph.osd.3.smithi062.stderr:2024-04-18T10:05:44.599+0000 7fbdc4692640 -1 osd.3 0 waiting for initial osdmap 2024-04-18T10:05:44.609 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-04-18T10:05:44.611+0000 7f3f299ef640 -1 osd.0 0 waiting for initial osdmap 2024-04-18T10:05:44.622 INFO:tasks.ceph.osd.1.smithi062.stderr:2024-04-18T10:05:44.623+0000 7fc688588640 -1 osd.1 0 waiting for initial osdmap 2024-04-18T10:05:44.624 INFO:tasks.ceph.osd.7.smithi145.stderr:2024-04-18T10:05:44.624+0000 7f14fdc35640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T10:05:44.631 INFO:tasks.ceph.osd.3.smithi062.stderr:2024-04-18T10:05:44.635+0000 7fbdbc4c6640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T10:05:44.647 INFO:tasks.ceph.osd.0.smithi062.stderr:2024-04-18T10:05:44.651+0000 7f3f21823640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T10:05:44.659 INFO:tasks.ceph.osd.1.smithi062.stderr:2024-04-18T10:05:44.663+0000 7fc6803bc640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-18T10:05:44.724 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-18T10:05:44.843 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a321a6640 1 Processor -- start 2024-04-18T10:05:44.843 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a321a6640 1 -- start start 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a321a6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2a2c149590 0x7f2a2c149a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a321a6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2a2c14e5b0 0x7f2a2c1509a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a321a6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a2c0b6400 con 0x7f2a2c0b4c40 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a321a6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f2a2c073f30 con 0x7f2a2c149590 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a321a6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f2a2c074180 con 0x7f2a2c14e5b0 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a2affd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2a2c149590 0x7f2a2c149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a2affd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2a2c149590 0x7f2a2c149a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48618/0 (socket says 172.21.15.62:48618) 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a2affd640 1 -- 172.21.15.62:0/3089921683 learned_addr learned my addr 172.21.15.62:0/3089921683 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a2bfff640 1 --2- 172.21.15.62:0/3089921683 >> v2:172.21.15.145:3300/0 conn(0x7f2a2c14e5b0 0x7f2a2c1509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a2affd640 1 -- 172.21.15.62:0/3089921683 >> v2:172.21.15.145:3300/0 conn(0x7f2a2c14e5b0 msgr2=0x7f2a2c1509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a2affd640 1 --2- 172.21.15.62:0/3089921683 >> v2:172.21.15.145:3300/0 conn(0x7f2a2c14e5b0 0x7f2a2c1509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a2affd640 1 -- 172.21.15.62:0/3089921683 >> v1:172.21.15.145:6789/0 conn(0x7f2a2c0b4c40 legacy=0x7f2a2c0b5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:44.845 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.847+0000 7f2a2affd640 1 -- 172.21.15.62:0/3089921683 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2c09cac0 con 0x7f2a2c149590 2024-04-18T10:05:44.847 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2affd640 1 --2- 172.21.15.62:0/3089921683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c149590 0x7f2a2c149a00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c075a20 tx=0x7f2a1c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b359b7f04112022e server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:44.847 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2a7fc640 1 -- 172.21.15.62:0/3089921683 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c0ab610 con 0x7f2a2c149590 2024-04-18T10:05:44.847 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2a7fc640 1 -- 172.21.15.62:0/3089921683 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a1c0b0070 con 0x7f2a2c149590 2024-04-18T10:05:44.847 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2a7fc640 1 -- 172.21.15.62:0/3089921683 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c0ab9d0 con 0x7f2a2c149590 2024-04-18T10:05:44.848 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 -- 172.21.15.62:0/3089921683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c149590 msgr2=0x7f2a2c149a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:44.848 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 --2- 172.21.15.62:0/3089921683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c149590 0x7f2a2c149a00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c075a20 tx=0x7f2a1c01c520 comp rx=0 tx=0).stop 2024-04-18T10:05:44.848 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 -- 172.21.15.62:0/3089921683 shutdown_connections 2024-04-18T10:05:44.849 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 --2- 172.21.15.62:0/3089921683 >> v2:172.21.15.145:3300/0 conn(0x7f2a2c14e5b0 0x7f2a2c1509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:44.849 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 --2- 172.21.15.62:0/3089921683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c149590 0x7f2a2c149a00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:44.849 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 -- 172.21.15.62:0/3089921683 >> 172.21.15.62:0/3089921683 conn(0x7f2a2c0b9350 msgr2=0x7f2a2c0b9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:44.849 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 -- 172.21.15.62:0/3089921683 shutdown_connections 2024-04-18T10:05:44.849 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 -- 172.21.15.62:0/3089921683 wait complete. 2024-04-18T10:05:44.849 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 Processor -- start 2024-04-18T10:05:44.849 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 -- start start 2024-04-18T10:05:44.852 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a2c0b4c40 0x7f2a2c0b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:44.852 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c149590 0x7f2a2c0b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:44.852 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a2c14e5b0 0x7f2a2c1423b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:44.852 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c09c400 con 0x7f2a2c149590 2024-04-18T10:05:44.852 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f2a2c0b5020 con 0x7f2a2c0b4c40 2024-04-18T10:05:44.852 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a321a6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2a2c09c580 con 0x7f2a2c14e5b0 2024-04-18T10:05:44.852 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c149590 0x7f2a2c0b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:44.852 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c149590 0x7f2a2c0b26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48630/0 (socket says 172.21.15.62:48630) 2024-04-18T10:05:44.852 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2affd640 1 -- 172.21.15.62:0/1232265545 learned_addr learned my addr 172.21.15.62:0/1232265545 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:44.852 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2b7fe640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a2c0b4c40 0x7f2a2c0b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:44.852 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2bfff640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a2c14e5b0 0x7f2a2c1423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:44.852 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2affd640 1 -- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a2c14e5b0 msgr2=0x7f2a2c1423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:44.853 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2affd640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a2c14e5b0 0x7f2a2c1423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:44.853 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2affd640 1 -- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a2c0b4c40 msgr2=0x7f2a2c0b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:44.853 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2affd640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a2c0b4c40 0x7f2a2c0b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:44.853 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2affd640 1 -- 172.21.15.62:0/1232265545 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a2c0b0c10 con 0x7f2a2c149590 2024-04-18T10:05:44.853 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.851+0000 7f2a2bfff640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a2c14e5b0 0x7f2a2c1423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:05:44.853 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.855+0000 7f2a2affd640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c149590 0x7f2a2c0b26d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c019130 tx=0x7f2a1c01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:44.853 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.855+0000 7f2a13fff640 1 -- 172.21.15.62:0/1232265545 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c0be5f0 con 0x7f2a2c149590 2024-04-18T10:05:44.853 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.855+0000 7f2a13fff640 1 -- 172.21.15.62:0/1232265545 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a1c0be790 con 0x7f2a2c149590 2024-04-18T10:05:44.853 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.855+0000 7f2a13fff640 1 -- 172.21.15.62:0/1232265545 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c0beb40 con 0x7f2a2c149590 2024-04-18T10:05:44.853 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.855+0000 7f2a321a6640 1 -- 172.21.15.62:0/1232265545 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a2c0b31e0 con 0x7f2a2c149590 2024-04-18T10:05:44.853 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.855+0000 7f2a321a6640 1 -- 172.21.15.62:0/1232265545 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a2c0b1190 con 0x7f2a2c149590 2024-04-18T10:05:44.853 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.855+0000 7f2a321a6640 1 -- 172.21.15.62:0/1232265545 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a2c04fda0 con 0x7f2a2c149590 2024-04-18T10:05:44.854 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.855+0000 7f2a13fff640 1 -- 172.21.15.62:0/1232265545 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 4) ==== 87931+0+0 (secure 0 0 0) 0x7f2a1c0c73f0 con 0x7f2a2c149590 2024-04-18T10:05:44.854 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.855+0000 7f2a13fff640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2a180682f0 0x7f2a1806a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:44.854 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.855+0000 7f2a13fff640 1 -- 172.21.15.62:0/1232265545 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f2a1c120860 con 0x7f2a2c149590 2024-04-18T10:05:44.854 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.855+0000 7f2a2b7fe640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2a180682f0 0x7f2a1806a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:44.854 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.855+0000 7f2a2b7fe640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2a180682f0 0x7f2a1806a7b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2a20001650 tx=0x7f2a20009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:44.857 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:44.859+0000 7f2a13fff640 1 -- 172.21.15.62:0/1232265545 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2a1c0a8050 con 0x7f2a2c149590 2024-04-18T10:05:45.019 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.023+0000 7f2a321a6640 1 -- 172.21.15.62:0/1232265545 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a2c0b5ad0 con 0x7f2a2c149590 2024-04-18T10:05:45.021 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.023+0000 7f2a13fff640 1 -- 172.21.15.62:0/1232265545 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f2a1c016110 con 0x7f2a2c149590 2024-04-18T10:05:45.021 INFO:teuthology.misc.health.smithi062.stdout: 2024-04-18T10:05:45.021 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":12,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:05:42.378522+0000","last_up_change":"0.000000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:05:45.023 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.027+0000 7f2a321a6640 1 -- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2a180682f0 msgr2=0x7f2a1806a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:45.023 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.027+0000 7f2a321a6640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2a180682f0 0x7f2a1806a7b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2a20001650 tx=0x7f2a20009040 comp rx=0 tx=0).stop 2024-04-18T10:05:45.023 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.027+0000 7f2a321a6640 1 -- 172.21.15.62:0/1232265545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c149590 msgr2=0x7f2a2c0b26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:45.023 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.027+0000 7f2a321a6640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c149590 0x7f2a2c0b26d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c019130 tx=0x7f2a1c01ad60 comp rx=0 tx=0).stop 2024-04-18T10:05:45.023 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.027+0000 7f2a321a6640 1 -- 172.21.15.62:0/1232265545 shutdown_connections 2024-04-18T10:05:45.023 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.027+0000 7f2a321a6640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2a180682f0 0x7f2a1806a7b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:45.024 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.027+0000 7f2a321a6640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a2c14e5b0 0x7f2a2c1423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:45.024 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.027+0000 7f2a321a6640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a2c149590 0x7f2a2c0b26d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:45.024 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.027+0000 7f2a321a6640 1 --2- 172.21.15.62:0/1232265545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a2c0b4c40 0x7f2a2c0b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:45.024 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.027+0000 7f2a321a6640 1 -- 172.21.15.62:0/1232265545 >> 172.21.15.62:0/1232265545 conn(0x7f2a2c0b9350 msgr2=0x7f2a2c148aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:45.024 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.027+0000 7f2a321a6640 1 -- 172.21.15.62:0/1232265545 shutdown_connections 2024-04-18T10:05:45.024 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:45.027+0000 7f2a321a6640 1 -- 172.21.15.62:0/1232265545 wait complete. 2024-04-18T10:05:45.040 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-18T10:05:51.846 DEBUG:teuthology.orchestra.run.smithi062:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-18T10:05:51.973 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8f232640 1 Processor -- start 2024-04-18T10:05:51.973 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8f232640 1 -- start start 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8f232640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a880b4c40 0x7f6a880b5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8f232640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6a8814e5b0 0x7f6a881509a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8f232640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a8809cac0 con 0x7f6a88148580 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8f232640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f6a88073f30 con 0x7f6a8814e5b0 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8f232640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f6a88074180 con 0x7f6a880b4c40 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8d7cf640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6a8814e5b0 0x7f6a881509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8d7cf640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6a8814e5b0 0x7f6a881509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40532/0 (socket says 172.21.15.62:40532) 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8d7cf640 1 -- 172.21.15.62:0/690481159 learned_addr learned my addr 172.21.15.62:0/690481159 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8d7cf640 1 -- 172.21.15.62:0/690481159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a880b4c40 msgr2=0x7f6a880b5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8cfce640 1 --2- 172.21.15.62:0/690481159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a880b4c40 0x7f6a880b5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8d7cf640 1 --2- 172.21.15.62:0/690481159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a880b4c40 0x7f6a880b5cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8d7cf640 1 -- 172.21.15.62:0/690481159 >> v1:172.21.15.62:6789/0 conn(0x7f6a88148580 legacy=0x7f6a881489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:51.975 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a877fe640 1 -- 172.21.15.62:0/690481159 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3094292351 0 0) 0x7f6a8809cac0 con 0x7f6a88148580 2024-04-18T10:05:51.976 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8d7cf640 1 -- 172.21.15.62:0/690481159 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a8809d0f0 con 0x7f6a8814e5b0 2024-04-18T10:05:51.976 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.974+0000 7f6a8cfce640 1 --2- 172.21.15.62:0/690481159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a880b4c40 0x7f6a880b5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:05:51.977 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.978+0000 7f6a8d7cf640 1 --2- 172.21.15.62:0/690481159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a8814e5b0 0x7f6a881509a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009e50 tx=0x7f6a8001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a45ad7bbfed03b server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:51.977 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.978+0000 7f6a877fe640 1 -- 172.21.15.62:0/690481159 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a800a5700 con 0x7f6a8814e5b0 2024-04-18T10:05:51.977 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.978+0000 7f6a877fe640 1 -- 172.21.15.62:0/690481159 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a800a58a0 con 0x7f6a8814e5b0 2024-04-18T10:05:51.977 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.978+0000 7f6a877fe640 1 -- 172.21.15.62:0/690481159 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a800a5c70 con 0x7f6a8814e5b0 2024-04-18T10:05:51.978 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.978+0000 7f6a8f232640 1 -- 172.21.15.62:0/690481159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a8814e5b0 msgr2=0x7f6a881509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:51.978 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.978+0000 7f6a8f232640 1 --2- 172.21.15.62:0/690481159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a8814e5b0 0x7f6a881509a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009e50 tx=0x7f6a8001ca30 comp rx=0 tx=0).stop 2024-04-18T10:05:51.978 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.978+0000 7f6a8f232640 1 -- 172.21.15.62:0/690481159 shutdown_connections 2024-04-18T10:05:51.978 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.978+0000 7f6a8f232640 1 --2- 172.21.15.62:0/690481159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a8814e5b0 0x7f6a881509a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:51.978 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.978+0000 7f6a8f232640 1 --2- 172.21.15.62:0/690481159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a880b4c40 0x7f6a880b5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:51.978 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.978+0000 7f6a8f232640 1 -- 172.21.15.62:0/690481159 >> 172.21.15.62:0/690481159 conn(0x7f6a880b9350 msgr2=0x7f6a880b9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:51.978 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.978+0000 7f6a8f232640 1 -- 172.21.15.62:0/690481159 shutdown_connections 2024-04-18T10:05:51.978 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.978+0000 7f6a8f232640 1 -- 172.21.15.62:0/690481159 wait complete. 2024-04-18T10:05:51.979 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8f232640 1 Processor -- start 2024-04-18T10:05:51.979 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8f232640 1 -- start start 2024-04-18T10:05:51.981 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8f232640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a880b4c40 0x7f6a88072100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:51.981 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8f232640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a88148580 0x7f6a88072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:51.981 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8f232640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a8814e5b0 0x7f6a881e3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:51.981 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8f232640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8809c580 con 0x7f6a88148580 2024-04-18T10:05:51.981 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8f232640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8809c400 con 0x7f6a880b4c40 2024-04-18T10:05:51.981 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8f232640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f6a880b6230 con 0x7f6a8814e5b0 2024-04-18T10:05:51.981 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a87fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a88148580 0x7f6a88072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a87fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a88148580 0x7f6a88072a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40534/0 (socket says 172.21.15.62:40534) 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a87fff640 1 -- 172.21.15.62:0/176443776 learned_addr learned my addr 172.21.15.62:0/176443776 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8d7cf640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a8814e5b0 0x7f6a881e3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8cfce640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a880b4c40 0x7f6a88072100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a87fff640 1 -- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a8814e5b0 msgr2=0x7f6a881e3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a87fff640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a8814e5b0 0x7f6a881e3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a87fff640 1 -- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a880b4c40 msgr2=0x7f6a88072100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a87fff640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a880b4c40 0x7f6a88072100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a87fff640 1 -- 172.21.15.62:0/176443776 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a880affb0 con 0x7f6a88148580 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a87fff640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a88148580 0x7f6a88072a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00c4d0 tx=0x7f6a7c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a857fa640 1 -- 172.21.15.62:0/176443776 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c017070 con 0x7f6a88148580 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a857fa640 1 -- 172.21.15.62:0/176443776 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a7c005480 con 0x7f6a88148580 2024-04-18T10:05:51.982 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a857fa640 1 -- 172.21.15.62:0/176443776 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c003840 con 0x7f6a88148580 2024-04-18T10:05:51.983 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8f232640 1 -- 172.21.15.62:0/176443776 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a880b06c0 con 0x7f6a88148580 2024-04-18T10:05:51.983 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8f232640 1 -- 172.21.15.62:0/176443776 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a88070a10 con 0x7f6a88148580 2024-04-18T10:05:51.983 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.982+0000 7f6a8f232640 1 -- 172.21.15.62:0/176443776 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a8804fda0 con 0x7f6a88148580 2024-04-18T10:05:51.984 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.986+0000 7f6a857fa640 1 -- 172.21.15.62:0/176443776 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6a7c01a070 con 0x7f6a88148580 2024-04-18T10:05:51.984 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.986+0000 7f6a857fa640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6a74068380 0x7f6a7406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:51.984 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.986+0000 7f6a857fa640 1 -- 172.21.15.62:0/176443776 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f6a7c0859d0 con 0x7f6a88148580 2024-04-18T10:05:51.984 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.986+0000 7f6a8cfce640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6a74068380 0x7f6a7406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:51.984 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.986+0000 7f6a8cfce640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6a74068380 0x7f6a7406a840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800a9a0 tx=0x7f6a78008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:51.988 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:51.990+0000 7f6a857fa640 1 -- 172.21.15.62:0/176443776 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6a7c056430 con 0x7f6a88148580 2024-04-18T10:05:52.152 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.154+0000 7f6a8f232640 1 -- 172.21.15.62:0/176443776 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6a880b5ad0 con 0x7f6a88148580 2024-04-18T10:05:52.153 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.154+0000 7f6a857fa640 1 -- 172.21.15.62:0/176443776 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12872 (secure 0 0 0) 0x7f6a7c055d30 con 0x7f6a88148580 2024-04-18T10:05:52.154 INFO:teuthology.misc.health.smithi062.stdout: 2024-04-18T10:05:52.154 INFO:teuthology.misc.health.smithi062.stdout:{"epoch":17,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:05:49.531602+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:05:52.156 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.158+0000 7f6a8f232640 1 -- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6a74068380 msgr2=0x7f6a7406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.156 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.158+0000 7f6a8f232640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6a74068380 0x7f6a7406a840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800a9a0 tx=0x7f6a78008040 comp rx=0 tx=0).stop 2024-04-18T10:05:52.156 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.158+0000 7f6a8f232640 1 -- 172.21.15.62:0/176443776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a88148580 msgr2=0x7f6a88072a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.156 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.158+0000 7f6a8f232640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a88148580 0x7f6a88072a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00c4d0 tx=0x7f6a7c00c990 comp rx=0 tx=0).stop 2024-04-18T10:05:52.156 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.158+0000 7f6a8f232640 1 -- 172.21.15.62:0/176443776 shutdown_connections 2024-04-18T10:05:52.156 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.158+0000 7f6a8f232640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6a74068380 0x7f6a7406a840 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.156 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.158+0000 7f6a8f232640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6a8814e5b0 0x7f6a881e3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.156 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.158+0000 7f6a8f232640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6a88148580 0x7f6a88072a80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.156 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.158+0000 7f6a8f232640 1 --2- 172.21.15.62:0/176443776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6a880b4c40 0x7f6a88072100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.157 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.158+0000 7f6a8f232640 1 -- 172.21.15.62:0/176443776 >> 172.21.15.62:0/176443776 conn(0x7f6a880b9350 msgr2=0x7f6a8809afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:52.157 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.158+0000 7f6a8f232640 1 -- 172.21.15.62:0/176443776 shutdown_connections 2024-04-18T10:05:52.157 INFO:teuthology.misc.health.smithi062.stderr:2024-04-18T10:05:52.158+0000 7f6a8f232640 1 -- 172.21.15.62:0/176443776 wait complete. 2024-04-18T10:05:52.172 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-18T10:05:52.172 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-18T10:05:52.172 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-18T10:05:52.172 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-18T10:05:52.175 INFO:tasks.ceph.mds.a:Started 2024-04-18T10:05:52.175 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-18T10:05:52.175 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-18T10:05:52.178 INFO:tasks.ceph.mds.b:Started 2024-04-18T10:05:52.178 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-18T10:05:52.178 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:05:52.250 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-04-18T10:05:52.276 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-04-18T10:05:52.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fcd02d6a640 1 Processor -- start 2024-04-18T10:05:52.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fcd02d6a640 1 -- start start 2024-04-18T10:05:52.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fcd02d6a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fccfc0bbe40 0x7fccfc0b3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fcd02d6a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fccfc0b4100 0x7fccfc0b5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fcd02d6a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccfc0bc220 con 0x7fccfc072620 2024-04-18T10:05:52.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fcd02d6a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fccfc0b3990 con 0x7fccfc0b4100 2024-04-18T10:05:52.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fcd02d6a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fccfc0b2b00 con 0x7fccfc0bbe40 2024-04-18T10:05:52.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fccfbfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fccfc0b4100 0x7fccfc0b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fccfbfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fccfc0b4100 0x7fccfc0b5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40558/0 (socket says 172.21.15.62:40558) 2024-04-18T10:05:52.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fccfbfff640 1 -- 172.21.15.62:0/4178032163 learned_addr learned my addr 172.21.15.62:0/4178032163 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:52.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fccfbfff640 1 -- 172.21.15.62:0/4178032163 >> v2:172.21.15.145:3300/0 conn(0x7fccfc0bbe40 msgr2=0x7fccfc0b3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:52.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fcd00b06640 1 --2- 172.21.15.62:0/4178032163 >> v2:172.21.15.145:3300/0 conn(0x7fccfc0bbe40 0x7fccfc0b3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fccfbfff640 1 --2- 172.21.15.62:0/4178032163 >> v2:172.21.15.145:3300/0 conn(0x7fccfc0bbe40 0x7fccfc0b3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fccfbfff640 1 -- 172.21.15.62:0/4178032163 >> v1:172.21.15.62:6789/0 conn(0x7fccfc072620 legacy=0x7fccfc072a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fccfbfff640 1 -- 172.21.15.62:0/4178032163 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccfc0aefc0 con 0x7fccfc0b4100 2024-04-18T10:05:52.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fccfb7fe640 1 -- 172.21.15.62:0/4178032163 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2406945513 0 0) 0x7fccfc0bc220 con 0x7fccfc072620 2024-04-18T10:05:52.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.298+0000 7fcd00b06640 1 --2- 172.21.15.62:0/4178032163 >> v2:172.21.15.145:3300/0 conn(0x7fccfc0bbe40 0x7fccfc0b3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:05:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fccfbfff640 1 --2- 172.21.15.62:0/4178032163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccfc0b4100 0x7fccfc0b5420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fccfc073830 tx=0x7fcce801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10335814f52854a server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fccfb7fe640 1 -- 172.21.15.62:0/4178032163 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce80b0070 con 0x7fccfc0b4100 2024-04-18T10:05:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fccfb7fe640 1 -- 172.21.15.62:0/4178032163 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcce80ab600 con 0x7fccfc0b4100 2024-04-18T10:05:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fccfb7fe640 1 -- 172.21.15.62:0/4178032163 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce80ab9d0 con 0x7fccfc0b4100 2024-04-18T10:05:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 -- 172.21.15.62:0/4178032163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccfc0b4100 msgr2=0x7fccfc0b5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 --2- 172.21.15.62:0/4178032163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccfc0b4100 0x7fccfc0b5420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fccfc073830 tx=0x7fcce801c490 comp rx=0 tx=0).stop 2024-04-18T10:05:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 -- 172.21.15.62:0/4178032163 shutdown_connections 2024-04-18T10:05:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 --2- 172.21.15.62:0/4178032163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccfc0b4100 0x7fccfc0b5420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 --2- 172.21.15.62:0/4178032163 >> v2:172.21.15.145:3300/0 conn(0x7fccfc0bbe40 0x7fccfc0b3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 -- 172.21.15.62:0/4178032163 >> 172.21.15.62:0/4178032163 conn(0x7fccfc067e10 msgr2=0x7fccfc0bb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 -- 172.21.15.62:0/4178032163 shutdown_connections 2024-04-18T10:05:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 -- 172.21.15.62:0/4178032163 wait complete. 2024-04-18T10:05:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 Processor -- start 2024-04-18T10:05:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 -- start start 2024-04-18T10:05:52.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccfc072620 0x7fccfc0aba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccfc0b4100 0x7fccfc0abf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccfc0bbe40 0x7fccfc141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc153470 con 0x7fccfc0bbe40 2024-04-18T10:05:52.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc1532f0 con 0x7fccfc0b4100 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.302+0000 7fcd02d6a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fccfc1535f0 con 0x7fccfc072620 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd01307640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccfc0bbe40 0x7fccfc141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd01307640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccfc0bbe40 0x7fccfc141ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40570/0 (socket says 172.21.15.62:40570) 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd01307640 1 -- 172.21.15.62:0/3478889161 learned_addr learned my addr 172.21.15.62:0/3478889161 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fccfbfff640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccfc0b4100 0x7fccfc0abf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd00b06640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccfc072620 0x7fccfc0aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd01307640 1 -- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccfc072620 msgr2=0x7fccfc0aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd01307640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccfc072620 0x7fccfc0aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd01307640 1 -- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccfc0b4100 msgr2=0x7fccfc0abf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd01307640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccfc0b4100 0x7fccfc0abf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd01307640 1 -- 172.21.15.62:0/3478889161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccfc0ad0f0 con 0x7fccfc0bbe40 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fccfbfff640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccfc0b4100 0x7fccfc0abf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd00b06640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccfc072620 0x7fccfc0aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd01307640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccfc0bbe40 0x7fccfc141ba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fccf400c780 tx=0x7fccf400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fccf97fa640 1 -- 172.21.15.62:0/3478889161 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4003db0 con 0x7fccfc0bbe40 2024-04-18T10:05:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fccf97fa640 1 -- 172.21.15.62:0/3478889161 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccf4005070 con 0x7fccfc0bbe40 2024-04-18T10:05:52.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd02d6a640 1 -- 172.21.15.62:0/3478889161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccfc0a9910 con 0x7fccfc0bbe40 2024-04-18T10:05:52.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fccf97fa640 1 -- 172.21.15.62:0/3478889161 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf4005460 con 0x7fccfc0bbe40 2024-04-18T10:05:52.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd02d6a640 1 -- 172.21.15.62:0/3478889161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccfc0ae460 con 0x7fccfc0bbe40 2024-04-18T10:05:52.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd02d6a640 1 -- 172.21.15.62:0/3478889161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccfc069230 con 0x7fccfc0bbe40 2024-04-18T10:05:52.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fccf97fa640 1 -- 172.21.15.62:0/3478889161 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fccf4005600 con 0x7fccfc0bbe40 2024-04-18T10:05:52.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fccf97fa640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcce0068380 0x7fcce006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fccf97fa640 1 -- 172.21.15.62:0/3478889161 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fccf40865d0 con 0x7fccfc0bbe40 2024-04-18T10:05:52.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd00b06640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcce0068380 0x7fcce006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.306+0000 7fcd00b06640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcce0068380 0x7fcce006a840 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fccf0008890 tx=0x7fccf0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:52.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.310+0000 7fccf97fa640 1 -- 172.21.15.62:0/3478889161 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fccf4056f70 con 0x7fccfc0bbe40 2024-04-18T10:05:52.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.474+0000 7fcd02d6a640 1 -- 172.21.15.62:0/3478889161 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fccfc143100 con 0x7fccfc0bbe40 2024-04-18T10:05:52.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.474+0000 7fccf97fa640 1 -- 172.21.15.62:0/3478889161 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12872 (secure 0 0 0) 0x7fccf4056870 con 0x7fccfc0bbe40 2024-04-18T10:05:52.475 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:52.475 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":17,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:05:49.531602+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:05:52.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.478+0000 7fcd02d6a640 1 -- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcce0068380 msgr2=0x7fcce006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.478+0000 7fcd02d6a640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcce0068380 0x7fcce006a840 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fccf0008890 tx=0x7fccf0008040 comp rx=0 tx=0).stop 2024-04-18T10:05:52.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.478+0000 7fcd02d6a640 1 -- 172.21.15.62:0/3478889161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccfc0bbe40 msgr2=0x7fccfc141ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.478+0000 7fcd02d6a640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccfc0bbe40 0x7fccfc141ba0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fccf400c780 tx=0x7fccf400cc40 comp rx=0 tx=0).stop 2024-04-18T10:05:52.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.478+0000 7fcd02d6a640 1 -- 172.21.15.62:0/3478889161 shutdown_connections 2024-04-18T10:05:52.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.478+0000 7fcd02d6a640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcce0068380 0x7fcce006a840 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.478+0000 7fcd02d6a640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fccfc0bbe40 0x7fccfc141ba0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.478+0000 7fcd02d6a640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccfc0b4100 0x7fccfc0abf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.478+0000 7fcd02d6a640 1 --2- 172.21.15.62:0/3478889161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccfc072620 0x7fccfc0aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.478+0000 7fcd02d6a640 1 -- 172.21.15.62:0/3478889161 >> 172.21.15.62:0/3478889161 conn(0x7fccfc067e10 msgr2=0x7fccfc0ba9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:52.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.478+0000 7fcd02d6a640 1 -- 172.21.15.62:0/3478889161 shutdown_connections 2024-04-18T10:05:52.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.478+0000 7fcd02d6a640 1 -- 172.21.15.62:0/3478889161 wait complete. 2024-04-18T10:05:52.495 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:05:52.495 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:05:52.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f9b56640 1 Processor -- start 2024-04-18T10:05:52.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f9b56640 1 -- start start 2024-04-18T10:05:52.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f9b56640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f84f406cbe0 0x7f84f40b69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f9b56640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f84f41419f0 0x7f84f4141e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f9b56640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84f406e420 con 0x7f84f40bbfb0 2024-04-18T10:05:52.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f9b56640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f84f4142370 con 0x7f84f406cbe0 2024-04-18T10:05:52.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f9b56640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f84f406e700 con 0x7f84f41419f0 2024-04-18T10:05:52.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f2ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f84f406cbe0 0x7f84f40b69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f2ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f84f406cbe0 0x7f84f40b69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40572/0 (socket says 172.21.15.62:40572) 2024-04-18T10:05:52.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f2ffd640 1 -- 172.21.15.62:0/3332267075 learned_addr learned my addr 172.21.15.62:0/3332267075 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:52.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f3fff640 1 --2- 172.21.15.62:0/3332267075 >> v2:172.21.15.145:3300/0 conn(0x7f84f41419f0 0x7f84f4141e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f2ffd640 1 -- 172.21.15.62:0/3332267075 >> v2:172.21.15.145:3300/0 conn(0x7f84f41419f0 msgr2=0x7f84f4141e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:52.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f2ffd640 1 --2- 172.21.15.62:0/3332267075 >> v2:172.21.15.145:3300/0 conn(0x7f84f41419f0 0x7f84f4141e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.620 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.618+0000 7f84f2ffd640 1 -- 172.21.15.62:0/3332267075 >> v1:172.21.15.145:6789/0 conn(0x7f84f40bbfb0 legacy=0x7f84f40b7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:52.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f2ffd640 1 -- 172.21.15.62:0/3332267075 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f40bc390 con 0x7f84f406cbe0 2024-04-18T10:05:52.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f2ffd640 1 --2- 172.21.15.62:0/3332267075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84f406cbe0 0x7f84f40b69b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f84f40b77e0 tx=0x7f84e801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18d45c29d8b7509 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:52.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f27fc640 1 -- 172.21.15.62:0/3332267075 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e80b0070 con 0x7f84f406cbe0 2024-04-18T10:05:52.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f27fc640 1 -- 172.21.15.62:0/3332267075 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84e80a4e50 con 0x7f84f406cbe0 2024-04-18T10:05:52.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f27fc640 1 -- 172.21.15.62:0/3332267075 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e80ab820 con 0x7f84f406cbe0 2024-04-18T10:05:52.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f9b56640 1 -- 172.21.15.62:0/3332267075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84f406cbe0 msgr2=0x7f84f40b69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f9b56640 1 --2- 172.21.15.62:0/3332267075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84f406cbe0 0x7f84f40b69b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f84f40b77e0 tx=0x7f84e801c3c0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f9b56640 1 -- 172.21.15.62:0/3332267075 shutdown_connections 2024-04-18T10:05:52.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f9b56640 1 --2- 172.21.15.62:0/3332267075 >> v2:172.21.15.145:3300/0 conn(0x7f84f41419f0 0x7f84f4141e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f9b56640 1 --2- 172.21.15.62:0/3332267075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84f406cbe0 0x7f84f40b69b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f9b56640 1 -- 172.21.15.62:0/3332267075 >> 172.21.15.62:0/3332267075 conn(0x7f84f4067ea0 msgr2=0x7f84f40bb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:52.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f9b56640 1 -- 172.21.15.62:0/3332267075 shutdown_connections 2024-04-18T10:05:52.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.622+0000 7f84f9b56640 1 -- 172.21.15.62:0/3332267075 wait complete. 2024-04-18T10:05:52.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f9b56640 1 Processor -- start 2024-04-18T10:05:52.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f9b56640 1 -- start start 2024-04-18T10:05:52.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f9b56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84f406cbe0 0x7f84f41e44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f9b56640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84f40bbfb0 0x7f84f41e03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f9b56640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84f41419f0 0x7f84f4144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f9b56640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f84f4154f70 con 0x7f84f406cbe0 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f9b56640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f84f41550f0 con 0x7f84f40bbfb0 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f9b56640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f84f4155270 con 0x7f84f41419f0 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84f406cbe0 0x7f84f41e44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84f406cbe0 0x7f84f41e44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40588/0 (socket says 172.21.15.62:40588) 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f37fe640 1 -- 172.21.15.62:0/2162172632 learned_addr learned my addr 172.21.15.62:0/2162172632 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f2ffd640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84f40bbfb0 0x7f84f41e03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f3fff640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84f41419f0 0x7f84f4144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f37fe640 1 -- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84f41419f0 msgr2=0x7f84f4144e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f37fe640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84f41419f0 0x7f84f4144e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f37fe640 1 -- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84f40bbfb0 msgr2=0x7f84f41e03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f37fe640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84f40bbfb0 0x7f84f41e03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f37fe640 1 -- 172.21.15.62:0/2162172632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f414e2a0 con 0x7f84f406cbe0 2024-04-18T10:05:52.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f2ffd640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84f40bbfb0 0x7f84f41e03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:05:52.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f37fe640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84f406cbe0 0x7f84f41e44b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f84e400c480 tx=0x7f84e400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:52.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84dbfff640 1 -- 172.21.15.62:0/2162172632 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4012830 con 0x7f84f406cbe0 2024-04-18T10:05:52.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84dbfff640 1 -- 172.21.15.62:0/2162172632 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84e4017070 con 0x7f84f406cbe0 2024-04-18T10:05:52.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f9b56640 1 -- 172.21.15.62:0/2162172632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f4144090 con 0x7f84f406cbe0 2024-04-18T10:05:52.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84dbfff640 1 -- 172.21.15.62:0/2162172632 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84e4012c30 con 0x7f84f406cbe0 2024-04-18T10:05:52.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f9b56640 1 -- 172.21.15.62:0/2162172632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f414c300 con 0x7f84f406cbe0 2024-04-18T10:05:52.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84f9b56640 1 -- 172.21.15.62:0/2162172632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84f4069230 con 0x7f84f406cbe0 2024-04-18T10:05:52.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84dbfff640 1 -- 172.21.15.62:0/2162172632 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f84e401c460 con 0x7f84f406cbe0 2024-04-18T10:05:52.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84dbfff640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f84e0068380 0x7f84e006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.626+0000 7f84dbfff640 1 -- 172.21.15.62:0/2162172632 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f84e40864a0 con 0x7f84f406cbe0 2024-04-18T10:05:52.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.630+0000 7f84f2ffd640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f84e0068380 0x7f84e006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.630+0000 7f84f2ffd640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f84e0068380 0x7f84e006a840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f84e80ad000 tx=0x7f84e8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:52.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.634+0000 7f84dbfff640 1 -- 172.21.15.62:0/2162172632 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f84e4056e40 con 0x7f84f406cbe0 2024-04-18T10:05:52.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.794+0000 7f84f9b56640 1 -- 172.21.15.62:0/2162172632 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f84f4146a20 con 0x7f84f406cbe0 2024-04-18T10:05:52.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.794+0000 7f84dbfff640 1 -- 172.21.15.62:0/2162172632 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f84e4056740 con 0x7f84f406cbe0 2024-04-18T10:05:52.794 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:05:52.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.798+0000 7f84f9b56640 1 -- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f84e0068380 msgr2=0x7f84e006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.798+0000 7f84f9b56640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f84e0068380 0x7f84e006a840 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f84e80ad000 tx=0x7f84e8008040 comp rx=0 tx=0).stop 2024-04-18T10:05:52.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.798+0000 7f84f9b56640 1 -- 172.21.15.62:0/2162172632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84f406cbe0 msgr2=0x7f84f41e44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.798+0000 7f84f9b56640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84f406cbe0 0x7f84f41e44b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f84e400c480 tx=0x7f84e400c940 comp rx=0 tx=0).stop 2024-04-18T10:05:52.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.798+0000 7f84f9b56640 1 -- 172.21.15.62:0/2162172632 shutdown_connections 2024-04-18T10:05:52.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.798+0000 7f84f9b56640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f84e0068380 0x7f84e006a840 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.798+0000 7f84f9b56640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84f41419f0 0x7f84f4144e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.798+0000 7f84f9b56640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84f40bbfb0 0x7f84f41e03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.798+0000 7f84f9b56640 1 --2- 172.21.15.62:0/2162172632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84f406cbe0 0x7f84f41e44b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.798+0000 7f84f9b56640 1 -- 172.21.15.62:0/2162172632 >> 172.21.15.62:0/2162172632 conn(0x7f84f4067ea0 msgr2=0x7f84f40b9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:52.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.798+0000 7f84f9b56640 1 -- 172.21.15.62:0/2162172632 shutdown_connections 2024-04-18T10:05:52.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.798+0000 7f84f9b56640 1 -- 172.21.15.62:0/2162172632 wait complete. 2024-04-18T10:05:52.813 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:05:52.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.938+0000 7fdbf0156640 1 Processor -- start 2024-04-18T10:05:52.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.938+0000 7fdbf0156640 1 -- start start 2024-04-18T10:05:52.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbf0156640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbe809c4d0 0x7fdbe809dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbf0156640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdbe814c080 0x7fdbe8145e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbf0156640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbe80ba170 con 0x7fdbe80b9d90 2024-04-18T10:05:52.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbf0156640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fdbe814f810 con 0x7fdbe814c080 2024-04-18T10:05:52.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbf0156640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fdbe814faf0 con 0x7fdbe809c4d0 2024-04-18T10:05:52.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbee6f3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdbe814c080 0x7fdbe8145e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbee6f3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdbe814c080 0x7fdbe8145e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40602/0 (socket says 172.21.15.62:40602) 2024-04-18T10:05:52.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbee6f3640 1 -- 172.21.15.62:0/530978231 learned_addr learned my addr 172.21.15.62:0/530978231 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:52.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbed6f1640 1 --2- 172.21.15.62:0/530978231 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbe809c4d0 0x7fdbe809dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbee6f3640 1 -- 172.21.15.62:0/530978231 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbe809c4d0 msgr2=0x7fdbe809dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:52.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbee6f3640 1 --2- 172.21.15.62:0/530978231 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbe809c4d0 0x7fdbe809dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbee6f3640 1 -- 172.21.15.62:0/530978231 >> v1:172.21.15.145:6789/0 conn(0x7fdbe80b9d90 legacy=0x7fdbe8150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbee6f3640 1 -- 172.21.15.62:0/530978231 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe809e160 con 0x7fdbe814c080 2024-04-18T10:05:52.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbee6f3640 1 --2- 172.21.15.62:0/530978231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe814c080 0x7fdbe8145e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0009870 tx=0x7fdbe001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d431d624d7252d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:52.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbecef0640 1 -- 172.21.15.62:0/530978231 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe00b2070 con 0x7fdbe814c080 2024-04-18T10:05:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbecef0640 1 -- 172.21.15.62:0/530978231 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbe00a9070 con 0x7fdbe814c080 2024-04-18T10:05:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.942+0000 7fdbecef0640 1 -- 172.21.15.62:0/530978231 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbe00ad880 con 0x7fdbe814c080 2024-04-18T10:05:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 -- 172.21.15.62:0/530978231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe814c080 msgr2=0x7fdbe8145e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 --2- 172.21.15.62:0/530978231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe814c080 0x7fdbe8145e00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fdbe0009870 tx=0x7fdbe001c560 comp rx=0 tx=0).stop 2024-04-18T10:05:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 -- 172.21.15.62:0/530978231 shutdown_connections 2024-04-18T10:05:52.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 --2- 172.21.15.62:0/530978231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe814c080 0x7fdbe8145e00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 --2- 172.21.15.62:0/530978231 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbe809c4d0 0x7fdbe809dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 -- 172.21.15.62:0/530978231 >> 172.21.15.62:0/530978231 conn(0x7fdbe806c0d0 msgr2=0x7fdbe80b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 -- 172.21.15.62:0/530978231 shutdown_connections 2024-04-18T10:05:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 -- 172.21.15.62:0/530978231 wait complete. 2024-04-18T10:05:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 Processor -- start 2024-04-18T10:05:52.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 -- start start 2024-04-18T10:05:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe809c4d0 0x7fdbe80b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdbe80b9d90 0x7fdbe81e4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbe814c080 0x7fdbe81e5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe8074020 con 0x7fdbe809c4d0 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fdbe8073ea0 con 0x7fdbe80b9d90 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fdbe8159a20 con 0x7fdbe814c080 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbedef2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe809c4d0 0x7fdbe80b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbedef2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe809c4d0 0x7fdbe80b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40612/0 (socket says 172.21.15.62:40612) 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbedef2640 1 -- 172.21.15.62:0/3821816203 learned_addr learned my addr 172.21.15.62:0/3821816203 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbee6f3640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbe814c080 0x7fdbe81e5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbed6f1640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdbe80b9d90 0x7fdbe81e4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbedef2640 1 -- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbe814c080 msgr2=0x7fdbe81e5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbedef2640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbe814c080 0x7fdbe81e5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbedef2640 1 -- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdbe80b9d90 msgr2=0x7fdbe81e4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbedef2640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdbe80b9d90 0x7fdbe81e4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbedef2640 1 -- 172.21.15.62:0/3821816203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbe80b1d20 con 0x7fdbe809c4d0 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbedef2640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe809c4d0 0x7fdbe80b0a80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00c420 tx=0x7fdbdc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbda7fc640 1 -- 172.21.15.62:0/3821816203 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc0052d0 con 0x7fdbe809c4d0 2024-04-18T10:05:52.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbda7fc640 1 -- 172.21.15.62:0/3821816203 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbdc005470 con 0x7fdbe809c4d0 2024-04-18T10:05:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 -- 172.21.15.62:0/3821816203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbe8144e70 con 0x7fdbe809c4d0 2024-04-18T10:05:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbda7fc640 1 -- 172.21.15.62:0/3821816203 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbdc007260 con 0x7fdbe809c4d0 2024-04-18T10:05:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.946+0000 7fdbf0156640 1 -- 172.21.15.62:0/3821816203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbe80b1f90 con 0x7fdbe809c4d0 2024-04-18T10:05:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.950+0000 7fdbf0156640 1 -- 172.21.15.62:0/3821816203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbe8068b90 con 0x7fdbe809c4d0 2024-04-18T10:05:52.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.950+0000 7fdbda7fc640 1 -- 172.21.15.62:0/3821816203 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdbdc007400 con 0x7fdbe809c4d0 2024-04-18T10:05:52.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.950+0000 7fdbda7fc640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdbcc068380 0x7fdbcc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:52.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.950+0000 7fdbda7fc640 1 -- 172.21.15.62:0/3821816203 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fdbdc085a30 con 0x7fdbe809c4d0 2024-04-18T10:05:52.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.950+0000 7fdbed6f1640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdbcc068380 0x7fdbcc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:52.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.950+0000 7fdbed6f1640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdbcc068380 0x7fdbcc06a840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4003fd0 tx=0x7fdbe40039a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:52.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:52.954+0000 7fdbda7fc640 1 -- 172.21.15.62:0/3821816203 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdbdc055d80 con 0x7fdbe809c4d0 2024-04-18T10:05:53.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.150+0000 7fdbf0156640 1 -- 172.21.15.62:0/3821816203 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdbe80b3850 con 0x7fdbe809c4d0 2024-04-18T10:05:53.150 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.150+0000 7fdbda7fc640 1 -- 172.21.15.62:0/3821816203 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7fdbdc05a080 con 0x7fdbe809c4d0 2024-04-18T10:05:53.150 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:53.150 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/2120622858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2120622858},{"type":"v1","addr":"172.21.15.62:6833","nonce":2120622858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-18T10:05:53.150 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 2 2024-04-18T10:05:53.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.154+0000 7fdbf0156640 1 -- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdbcc068380 msgr2=0x7fdbcc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:53.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.154+0000 7fdbf0156640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdbcc068380 0x7fdbcc06a840 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdbe4003fd0 tx=0x7fdbe40039a0 comp rx=0 tx=0).stop 2024-04-18T10:05:53.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.154+0000 7fdbf0156640 1 -- 172.21.15.62:0/3821816203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe809c4d0 msgr2=0x7fdbe80b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:53.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.154+0000 7fdbf0156640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe809c4d0 0x7fdbe80b0a80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc00c420 tx=0x7fdbdc00c8e0 comp rx=0 tx=0).stop 2024-04-18T10:05:53.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.154+0000 7fdbf0156640 1 -- 172.21.15.62:0/3821816203 shutdown_connections 2024-04-18T10:05:53.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.154+0000 7fdbf0156640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdbcc068380 0x7fdbcc06a840 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:53.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.154+0000 7fdbf0156640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdbe814c080 0x7fdbe81e5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:53.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.154+0000 7fdbf0156640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdbe80b9d90 0x7fdbe81e4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:53.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.154+0000 7fdbf0156640 1 --2- 172.21.15.62:0/3821816203 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbe809c4d0 0x7fdbe80b0a80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:53.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.154+0000 7fdbf0156640 1 -- 172.21.15.62:0/3821816203 >> 172.21.15.62:0/3821816203 conn(0x7fdbe806c0d0 msgr2=0x7fdbe80b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:53.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.154+0000 7fdbf0156640 1 -- 172.21.15.62:0/3821816203 shutdown_connections 2024-04-18T10:05:53.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:53.154+0000 7fdbf0156640 1 -- 172.21.15.62:0/3821816203 wait complete. 2024-04-18T10:05:55.170 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:05:55.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.302+0000 7f7b04605640 1 Processor -- start 2024-04-18T10:05:55.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b04605640 1 -- start start 2024-04-18T10:05:55.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b04605640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7afc0b6260 0x7f7afc148260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b04605640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7afc1487a0 0x7f7afc1419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b04605640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7afc0b6640 0x7f7afc0b5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b04605640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f7afc148cb0 con 0x7f7afc0b6260 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b04605640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f7afc0b4f90 con 0x7f7afc1487a0 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b04605640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7afc0b7850 con 0x7f7afc0b6640 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b023a1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7afc0b6260 0x7f7afc148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b023a1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7afc0b6260 0x7f7afc148260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40626/0 (socket says 172.21.15.62:40626) 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b023a1640 1 -- 172.21.15.62:0/2406969814 learned_addr learned my addr 172.21.15.62:0/2406969814 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b01ba0640 1 --2- 172.21.15.62:0/2406969814 >> v2:172.21.15.145:3300/0 conn(0x7f7afc1487a0 0x7f7afc1419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b02ba2640 1 --2- 172.21.15.62:0/2406969814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7afc0b6640 0x7f7afc0b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b023a1640 1 -- 172.21.15.62:0/2406969814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7afc0b6640 msgr2=0x7f7afc0b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b023a1640 1 --2- 172.21.15.62:0/2406969814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7afc0b6640 0x7f7afc0b5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b023a1640 1 -- 172.21.15.62:0/2406969814 >> v2:172.21.15.145:3300/0 conn(0x7f7afc1487a0 msgr2=0x7f7afc1419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:55.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b023a1640 1 --2- 172.21.15.62:0/2406969814 >> v2:172.21.15.145:3300/0 conn(0x7f7afc1487a0 0x7f7afc1419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b023a1640 1 -- 172.21.15.62:0/2406969814 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7afc0b4c20 con 0x7f7afc0b6260 2024-04-18T10:05:55.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b023a1640 1 --2- 172.21.15.62:0/2406969814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc0b6260 0x7f7afc148260 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7afc131660 tx=0x7f7af801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20e994b70ad398 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:55.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.306+0000 7f7b0139f640 1 -- 172.21.15.62:0/2406969814 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af80b1070 con 0x7f7afc0b6260 2024-04-18T10:05:55.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b0139f640 1 -- 172.21.15.62:0/2406969814 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af80ac600 con 0x7f7afc0b6260 2024-04-18T10:05:55.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b0139f640 1 -- 172.21.15.62:0/2406969814 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af80ac9d0 con 0x7f7afc0b6260 2024-04-18T10:05:55.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 -- 172.21.15.62:0/2406969814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc0b6260 msgr2=0x7f7afc148260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 --2- 172.21.15.62:0/2406969814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc0b6260 0x7f7afc148260 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7afc131660 tx=0x7f7af801c400 comp rx=0 tx=0).stop 2024-04-18T10:05:55.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 -- 172.21.15.62:0/2406969814 shutdown_connections 2024-04-18T10:05:55.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 --2- 172.21.15.62:0/2406969814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7afc0b6640 0x7f7afc0b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 --2- 172.21.15.62:0/2406969814 >> v2:172.21.15.145:3300/0 conn(0x7f7afc1487a0 0x7f7afc1419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 --2- 172.21.15.62:0/2406969814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc0b6260 0x7f7afc148260 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 -- 172.21.15.62:0/2406969814 >> 172.21.15.62:0/2406969814 conn(0x7f7afc0bd5c0 msgr2=0x7f7afc0bd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 -- 172.21.15.62:0/2406969814 shutdown_connections 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 -- 172.21.15.62:0/2406969814 wait complete. 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 Processor -- start 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 -- start start 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc0b6260 0x7f7afc142750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7afc0b6640 0x7f7afc1433a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7afc1487a0 0x7f7afc1438e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7afc14bea0 con 0x7f7afc0b6260 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7afc14bd20 con 0x7f7afc0b6640 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7afc14c020 con 0x7f7afc1487a0 2024-04-18T10:05:55.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b023a1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc0b6260 0x7f7afc142750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b023a1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc0b6260 0x7f7afc142750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40634/0 (socket says 172.21.15.62:40634) 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b023a1640 1 -- 172.21.15.62:0/2894280169 learned_addr learned my addr 172.21.15.62:0/2894280169 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b01ba0640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7afc0b6640 0x7f7afc1433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b02ba2640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7afc1487a0 0x7f7afc1438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b023a1640 1 -- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7afc1487a0 msgr2=0x7f7afc1438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b023a1640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7afc1487a0 0x7f7afc1438e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b023a1640 1 -- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7afc0b6640 msgr2=0x7f7afc1433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b023a1640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7afc0b6640 0x7f7afc1433a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b023a1640 1 -- 172.21.15.62:0/2894280169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7afc1458a0 con 0x7f7afc0b6260 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b01ba0640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7afc0b6640 0x7f7afc1433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b023a1640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc0b6260 0x7f7afc142750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7af801aab0 tx=0x7f7af801ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7af2ffd640 1 -- 172.21.15.62:0/2894280169 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af80b1040 con 0x7f7afc0b6260 2024-04-18T10:05:55.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7af2ffd640 1 -- 172.21.15.62:0/2894280169 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af80bc070 con 0x7f7afc0b6260 2024-04-18T10:05:55.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 -- 172.21.15.62:0/2894280169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7afc144a10 con 0x7f7afc0b6260 2024-04-18T10:05:55.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7af2ffd640 1 -- 172.21.15.62:0/2894280169 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af80b7850 con 0x7f7afc0b6260 2024-04-18T10:05:55.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.310+0000 7f7b04605640 1 -- 172.21.15.62:0/2894280169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7afc13f4f0 con 0x7f7afc0b6260 2024-04-18T10:05:55.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.314+0000 7f7b04605640 1 -- 172.21.15.62:0/2894280169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7afc073bf0 con 0x7f7afc0b6260 2024-04-18T10:05:55.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.314+0000 7f7af2ffd640 1 -- 172.21.15.62:0/2894280169 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7af80ac4c0 con 0x7f7afc0b6260 2024-04-18T10:05:55.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.314+0000 7f7af2ffd640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ae00683b0 0x7f7ae006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.314+0000 7f7af2ffd640 1 -- 172.21.15.62:0/2894280169 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f7af8120850 con 0x7f7afc0b6260 2024-04-18T10:05:55.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.314+0000 7f7b01ba0640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ae00683b0 0x7f7ae006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.314+0000 7f7b01ba0640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ae00683b0 0x7f7ae006a870 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0b55d0 tx=0x7f7aec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:55.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.318+0000 7f7af2ffd640 1 -- 172.21.15.62:0/2894280169 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7af8016110 con 0x7f7afc0b6260 2024-04-18T10:05:55.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 -- 172.21.15.62:0/2894280169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7afc1456a0 con 0x7f7afc0b6260 2024-04-18T10:05:55.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7af2ffd640 1 -- 172.21.15.62:0/2894280169 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2216 (secure 0 0 0) 0x7f7af80f11f0 con 0x7f7afc0b6260 2024-04-18T10:05:55.512 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:55.513 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/904929673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":904929673},{"type":"v1","addr":"172.21.15.145:6835","nonce":904929673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4251,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/2120622858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2120622858},{"type":"v1","addr":"172.21.15.62:6833","nonce":2120622858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-18T10:05:55.513 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 3 2024-04-18T10:05:55.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 -- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ae00683b0 msgr2=0x7f7ae006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ae00683b0 0x7f7ae006a870 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0b55d0 tx=0x7f7aec008040 comp rx=0 tx=0).stop 2024-04-18T10:05:55.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 -- 172.21.15.62:0/2894280169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc0b6260 msgr2=0x7f7afc142750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc0b6260 0x7f7afc142750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7af801aab0 tx=0x7f7af801ac40 comp rx=0 tx=0).stop 2024-04-18T10:05:55.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 -- 172.21.15.62:0/2894280169 shutdown_connections 2024-04-18T10:05:55.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ae00683b0 0x7f7ae006a870 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7afc1487a0 0x7f7afc1438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7afc0b6640 0x7f7afc1433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 --2- 172.21.15.62:0/2894280169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7afc0b6260 0x7f7afc142750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 -- 172.21.15.62:0/2894280169 >> 172.21.15.62:0/2894280169 conn(0x7f7afc0bd5c0 msgr2=0x7f7afc0b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:55.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 -- 172.21.15.62:0/2894280169 shutdown_connections 2024-04-18T10:05:55.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.514+0000 7f7b04605640 1 -- 172.21.15.62:0/2894280169 wait complete. 2024-04-18T10:05:55.530 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:05:55.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f7047232640 1 Processor -- start 2024-04-18T10:05:55.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f7047232640 1 -- start start 2024-04-18T10:05:55.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f7047232640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f70400b9d90 0x7f70401450e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f7047232640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7040145620 con 0x7f70400b44e0 2024-04-18T10:05:55.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f7047232640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7040148f50 con 0x7f70400b4970 2024-04-18T10:05:55.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f7047232640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f70400b4de0 con 0x7f70400b9d90 2024-04-18T10:05:55.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703ffff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f70400b44e0 0x7f7040148130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58600/0 (socket says 172.21.15.62:58600) 2024-04-18T10:05:55.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703ffff640 1 -- 172.21.15.62:0/1395652145 learned_addr learned my addr 172.21.15.62:0/1395652145 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:05:55.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f7044fce640 1 --2- 172.21.15.62:0/1395652145 >> v2:172.21.15.145:3300/0 conn(0x7f70400b9d90 0x7f70401450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464143899 0 0) 0x7f7034001120 con 0x7f70400b44e0 2024-04-18T10:05:55.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7028003530 con 0x7f70400b44e0 2024-04-18T10:05:55.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1979296747 0 0) 0x7f7028003530 con 0x7f70400b44e0 2024-04-18T10:05:55.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 >> v2:172.21.15.145:3300/0 conn(0x7f70400b9d90 msgr2=0x7f70401450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:55.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 --2- 172.21.15.62:0/1395652145 >> v2:172.21.15.145:3300/0 conn(0x7f70400b9d90 0x7f70401450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 >> v1:172.21.15.145:6789/0 conn(0x7f70400b4970 legacy=0x7f7040148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70400ba170 con 0x7f70400b44e0 2024-04-18T10:05:55.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f7034001850 con 0x7f70400b44e0 2024-04-18T10:05:55.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f7044fce640 1 --2- 172.21.15.62:0/1395652145 >> v2:172.21.15.145:3300/0 conn(0x7f70400b9d90 0x7f70401450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:05:55.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 >> v1:172.21.15.62:6789/0 conn(0x7f70400b44e0 legacy=0x7f7040148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 --2- 172.21.15.62:0/1395652145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70280039c0 0x7f7028003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 --2- 172.21.15.62:0/1395652145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7028004810 0x7f7028004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 --2- 172.21.15.62:0/1395652145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70280051d0 0x7f702800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f70400b4de0 con 0x7f7028004810 2024-04-18T10:05:55.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f702800c170 con 0x7f70280051d0 2024-04-18T10:05:55.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f702800c370 con 0x7f70280039c0 2024-04-18T10:05:55.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.650+0000 7f703ffff640 1 --2- 172.21.15.62:0/1395652145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7028004810 0x7f7028004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f7044fce640 1 --2- 172.21.15.62:0/1395652145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70280039c0 0x7f7028003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f70457cf640 1 --2- 172.21.15.62:0/1395652145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70280051d0 0x7f702800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f703ffff640 1 -- 172.21.15.62:0/1395652145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70280039c0 msgr2=0x7f7028003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:55.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f703ffff640 1 --2- 172.21.15.62:0/1395652145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70280039c0 0x7f7028003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f703ffff640 1 -- 172.21.15.62:0/1395652145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70280051d0 msgr2=0x7f702800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:55.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f703ffff640 1 --2- 172.21.15.62:0/1395652145 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70280051d0 0x7f702800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f703ffff640 1 -- 172.21.15.62:0/1395652145 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7028003530 con 0x7f7028004810 2024-04-18T10:05:55.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f70457cf640 1 -- 172.21.15.62:0/1395652145 reap_dead start 2024-04-18T10:05:55.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f703ffff640 1 --2- 172.21.15.62:0/1395652145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7028004810 0x7f7028004c90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f703400c4c0 tx=0x7f703401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b43ea2a02757b59 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:55.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f703400c0e0 con 0x7f7028004810 2024-04-18T10:05:55.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70340a9070 con 0x7f7028004810 2024-04-18T10:05:55.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f703f7fe640 1 -- 172.21.15.62:0/1395652145 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034005d80 con 0x7f7028004810 2024-04-18T10:05:55.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f7047232640 1 -- 172.21.15.62:0/1395652145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7028004810 msgr2=0x7f7028004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f7047232640 1 --2- 172.21.15.62:0/1395652145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7028004810 0x7f7028004c90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f703400c4c0 tx=0x7f703401cca0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f7047232640 1 -- 172.21.15.62:0/1395652145 shutdown_connections 2024-04-18T10:05:55.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f7047232640 1 --2- 172.21.15.62:0/1395652145 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7028004810 0x7f7028004c90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f7047232640 1 -- 172.21.15.62:0/1395652145 >> 172.21.15.62:0/1395652145 conn(0x7f704006c0d0 msgr2=0x7f70400b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:55.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f7047232640 1 -- 172.21.15.62:0/1395652145 shutdown_connections 2024-04-18T10:05:55.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.654+0000 7f7047232640 1 -- 172.21.15.62:0/1395652145 wait complete. 2024-04-18T10:05:55.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f7047232640 1 Processor -- start 2024-04-18T10:05:55.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f7047232640 1 -- start start 2024-04-18T10:05:55.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f7047232640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028004810 0x7f7040072ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f7047232640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70400b0890 0x7f70401e0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f7047232640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70401e0d90 0x7f7040143f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f7047232640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f704009e3d0 con 0x7f70400b0890 2024-04-18T10:05:55.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f7047232640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7040145b80 con 0x7f70401e0d90 2024-04-18T10:05:55.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f7047232640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f70400b0d10 con 0x7f7028004810 2024-04-18T10:05:55.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f703ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70400b0890 0x7f70401e0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f703ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70400b0890 0x7f70401e0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40664/0 (socket says 172.21.15.62:40664) 2024-04-18T10:05:55.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f703ffff640 1 -- 172.21.15.62:0/298201757 learned_addr learned my addr 172.21.15.62:0/298201757 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f7044fce640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028004810 0x7f7040072ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f70457cf640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70401e0d90 0x7f7040143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f703ffff640 1 -- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028004810 msgr2=0x7f7040072ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f703ffff640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028004810 0x7f7040072ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f703ffff640 1 -- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70401e0d90 msgr2=0x7f7040143f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f703ffff640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70401e0d90 0x7f7040143f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f703ffff640 1 -- 172.21.15.62:0/298201757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70400b1e30 con 0x7f70400b0890 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f703ffff640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70400b0890 0x7f70401e0850 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7034001b50 tx=0x7f7028005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f703d7fa640 1 -- 172.21.15.62:0/298201757 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034001120 con 0x7f70400b0890 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f703d7fa640 1 -- 172.21.15.62:0/298201757 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70340047f0 con 0x7f70400b0890 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f703d7fa640 1 -- 172.21.15.62:0/298201757 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7034004ba0 con 0x7f70400b0890 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f7047232640 1 -- 172.21.15.62:0/298201757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70401d69c0 con 0x7f70400b0890 2024-04-18T10:05:55.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f7047232640 1 -- 172.21.15.62:0/298201757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70400b20a0 con 0x7f70400b0890 2024-04-18T10:05:55.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.658+0000 7f7047232640 1 -- 172.21.15.62:0/298201757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7040068b90 con 0x7f70400b0890 2024-04-18T10:05:55.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.662+0000 7f703d7fa640 1 -- 172.21.15.62:0/298201757 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f70340a5780 con 0x7f70400b0890 2024-04-18T10:05:55.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.662+0000 7f703d7fa640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f702c068380 0x7f702c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.662+0000 7f703d7fa640 1 -- 172.21.15.62:0/298201757 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f703411f980 con 0x7f70400b0890 2024-04-18T10:05:55.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.662+0000 7f7044fce640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f702c068380 0x7f702c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.662+0000 7f7044fce640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f702c068380 0x7f702c06a840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f704006e490 tx=0x7f7030008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:55.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.666+0000 7f703d7fa640 1 -- 172.21.15.62:0/298201757 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7040145620 con 0x7f70400b0890 2024-04-18T10:05:55.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.830+0000 7f7047232640 1 -- 172.21.15.62:0/298201757 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70401d2b80 con 0x7f70400b0890 2024-04-18T10:05:55.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.830+0000 7f703d7fa640 1 -- 172.21.15.62:0/298201757 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12872 (secure 0 0 0) 0x7f70340f03e0 con 0x7f70400b0890 2024-04-18T10:05:55.830 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:55.830 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":17,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:05:49.531602+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:05:55.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.834+0000 7f7047232640 1 -- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f702c068380 msgr2=0x7f702c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.834+0000 7f7047232640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f702c068380 0x7f702c06a840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f704006e490 tx=0x7f7030008040 comp rx=0 tx=0).stop 2024-04-18T10:05:55.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.834+0000 7f7047232640 1 -- 172.21.15.62:0/298201757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70400b0890 msgr2=0x7f70401e0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.834+0000 7f7047232640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70400b0890 0x7f70401e0850 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7034001b50 tx=0x7f7028005840 comp rx=0 tx=0).stop 2024-04-18T10:05:55.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.834+0000 7f7047232640 1 -- 172.21.15.62:0/298201757 shutdown_connections 2024-04-18T10:05:55.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.834+0000 7f7047232640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f702c068380 0x7f702c06a840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.834+0000 7f7047232640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70401e0d90 0x7f7040143f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.834+0000 7f7047232640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70400b0890 0x7f70401e0850 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.834+0000 7f7047232640 1 --2- 172.21.15.62:0/298201757 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7028004810 0x7f7040072ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.834+0000 7f7047232640 1 -- 172.21.15.62:0/298201757 >> 172.21.15.62:0/298201757 conn(0x7f704006c0d0 msgr2=0x7f70401d30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:55.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.834+0000 7f7047232640 1 -- 172.21.15.62:0/298201757 shutdown_connections 2024-04-18T10:05:55.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.834+0000 7f7047232640 1 -- 172.21.15.62:0/298201757 wait complete. 2024-04-18T10:05:55.849 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:05:55.850 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:05:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5f2a6640 1 Processor -- start 2024-04-18T10:05:55.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5f2a6640 1 -- start start 2024-04-18T10:05:55.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5f2a6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3a5806d680 0x7f3a5814be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5f2a6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a580b5b40 con 0x7f3a5806d2a0 2024-04-18T10:05:55.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5f2a6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a580ae960 con 0x7f3a5806e570 2024-04-18T10:05:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5f2a6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f3a580ae680 con 0x7f3a5806d680 2024-04-18T10:05:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5d042640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3a5806d2a0 0x7f3a580aedf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58614/0 (socket says 172.21.15.62:58614) 2024-04-18T10:05:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5d042640 1 -- 172.21.15.62:0/950863566 learned_addr learned my addr 172.21.15.62:0/950863566 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:05:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5d843640 1 --2- 172.21.15.62:0/950863566 >> v2:172.21.15.145:3300/0 conn(0x7f3a5806d680 0x7f3a5814be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2929827745 0 0) 0x7f3a580b5b40 con 0x7f3a5806d2a0 2024-04-18T10:05:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a44003530 con 0x7f3a5806d2a0 2024-04-18T10:05:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 409723561 0 0) 0x7f3a44003530 con 0x7f3a5806d2a0 2024-04-18T10:05:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 >> v2:172.21.15.145:3300/0 conn(0x7f3a5806d680 msgr2=0x7f3a5814be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 --2- 172.21.15.62:0/950863566 >> v2:172.21.15.145:3300/0 conn(0x7f3a5806d680 0x7f3a5814be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 >> v1:172.21.15.145:6789/0 conn(0x7f3a5806e570 legacy=0x7f3a5814b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5809ea30 con 0x7f3a5806d2a0 2024-04-18T10:05:55.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f3a54001680 con 0x7f3a5806d2a0 2024-04-18T10:05:55.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 >> v1:172.21.15.62:6789/0 conn(0x7f3a5806d2a0 legacy=0x7f3a580aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 --2- 172.21.15.62:0/950863566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a440039c0 0x7f3a44003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 --2- 172.21.15.62:0/950863566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a44004810 0x7f3a44004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 --2- 172.21.15.62:0/950863566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a440051d0 0x7f3a4400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3a580ae680 con 0x7f3a440051d0 2024-04-18T10:05:55.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4400c170 con 0x7f3a44004810 2024-04-18T10:05:55.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3a4400c370 con 0x7f3a440039c0 2024-04-18T10:05:55.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5d843640 1 --2- 172.21.15.62:0/950863566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a440051d0 0x7f3a4400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5c841640 1 --2- 172.21.15.62:0/950863566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a44004810 0x7f3a44004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5d042640 1 --2- 172.21.15.62:0/950863566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a440039c0 0x7f3a44003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5d843640 1 -- 172.21.15.62:0/950863566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a440039c0 msgr2=0x7f3a44003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:55.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5d843640 1 --2- 172.21.15.62:0/950863566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a440039c0 0x7f3a44003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5d843640 1 -- 172.21.15.62:0/950863566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a44004810 msgr2=0x7f3a44004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:55.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5d843640 1 --2- 172.21.15.62:0/950863566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a44004810 0x7f3a44004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.970+0000 7f3a5d843640 1 -- 172.21.15.62:0/950863566 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a44003530 con 0x7f3a440051d0 2024-04-18T10:05:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5d843640 1 -- 172.21.15.62:0/950863566 reap_dead start 2024-04-18T10:05:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5d843640 1 --2- 172.21.15.62:0/950863566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a440051d0 0x7f3a4400b9d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009f90 tx=0x7f3a5001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7ff5288b73a973 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a50009c10 con 0x7f3a440051d0 2024-04-18T10:05:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a50003ba0 con 0x7f3a440051d0 2024-04-18T10:05:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a4bfff640 1 -- 172.21.15.62:0/950863566 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a500a5980 con 0x7f3a440051d0 2024-04-18T10:05:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/950863566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a440051d0 msgr2=0x7f3a4400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 --2- 172.21.15.62:0/950863566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a440051d0 0x7f3a4400b9d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009f90 tx=0x7f3a5001cc80 comp rx=0 tx=0).stop 2024-04-18T10:05:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/950863566 shutdown_connections 2024-04-18T10:05:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 --2- 172.21.15.62:0/950863566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a440051d0 0x7f3a4400b9d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/950863566 >> 172.21.15.62:0/950863566 conn(0x7f3a580b9690 msgr2=0x7f3a580b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/950863566 shutdown_connections 2024-04-18T10:05:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/950863566 wait complete. 2024-04-18T10:05:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 Processor -- start 2024-04-18T10:05:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 -- start start 2024-04-18T10:05:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a440051d0 0x7f3a581d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a580b0a10 0x7f3a581cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a580b0e90 0x7f3a58071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3a58073e70 con 0x7f3a580b0e90 2024-04-18T10:05:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5806e070 con 0x7f3a580b0a10 2024-04-18T10:05:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5f2a6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3a58073ff0 con 0x7f3a440051d0 2024-04-18T10:05:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5d843640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a580b0e90 0x7f3a58071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5d843640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a580b0e90 0x7f3a58071660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40674/0 (socket says 172.21.15.62:40674) 2024-04-18T10:05:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5d843640 1 -- 172.21.15.62:0/1277527338 learned_addr learned my addr 172.21.15.62:0/1277527338 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5c841640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a580b0a10 0x7f3a581cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5d042640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a440051d0 0x7f3a581d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5d843640 1 -- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a440051d0 msgr2=0x7f3a581d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5d843640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a440051d0 0x7f3a581d09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5d843640 1 -- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a580b0a10 msgr2=0x7f3a581cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5d843640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a580b0a10 0x7f3a581cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5d843640 1 -- 172.21.15.62:0/1277527338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a580b0350 con 0x7f3a580b0e90 2024-04-18T10:05:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.974+0000 7f3a5d843640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a580b0e90 0x7f3a58071660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3a5001cab0 tx=0x7f3a50009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.978+0000 7f3a49ffb640 1 -- 172.21.15.62:0/1277527338 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a50004020 con 0x7f3a580b0e90 2024-04-18T10:05:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.978+0000 7f3a49ffb640 1 -- 172.21.15.62:0/1277527338 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a500be070 con 0x7f3a580b0e90 2024-04-18T10:05:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.978+0000 7f3a49ffb640 1 -- 172.21.15.62:0/1277527338 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a50004420 con 0x7f3a580b0e90 2024-04-18T10:05:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.978+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/1277527338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a581ca780 con 0x7f3a580b0e90 2024-04-18T10:05:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.978+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/1277527338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a580aedf0 con 0x7f3a580b0e90 2024-04-18T10:05:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.978+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/1277527338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a58068c00 con 0x7f3a580b0e90 2024-04-18T10:05:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.978+0000 7f3a49ffb640 1 -- 172.21.15.62:0/1277527338 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3a500045c0 con 0x7f3a580b0e90 2024-04-18T10:05:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.978+0000 7f3a49ffb640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3a3c0683b0 0x7f3a3c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:55.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.978+0000 7f3a49ffb640 1 -- 172.21.15.62:0/1277527338 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f3a50121b80 con 0x7f3a580b0e90 2024-04-18T10:05:55.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.978+0000 7f3a5d042640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3a3c0683b0 0x7f3a3c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:55.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.978+0000 7f3a5d042640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3a3c0683b0 0x7f3a3c06a870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3a54000c10 tx=0x7f3a54009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:55.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:55.982+0000 7f3a49ffb640 1 -- 172.21.15.62:0/1277527338 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3a50016990 con 0x7f3a580b0e90 2024-04-18T10:05:56.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.138+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/1277527338 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3a5809cda0 con 0x7f3a580b0e90 2024-04-18T10:05:56.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.138+0000 7f3a49ffb640 1 -- 172.21.15.62:0/1277527338 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f3a500f2520 con 0x7f3a580b0e90 2024-04-18T10:05:56.137 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:05:56.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.138+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3a3c0683b0 msgr2=0x7f3a3c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:56.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.138+0000 7f3a5f2a6640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3a3c0683b0 0x7f3a3c06a870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3a54000c10 tx=0x7f3a54009040 comp rx=0 tx=0).stop 2024-04-18T10:05:56.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.138+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/1277527338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a580b0e90 msgr2=0x7f3a58071660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:56.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.138+0000 7f3a5f2a6640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a580b0e90 0x7f3a58071660 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3a5001cab0 tx=0x7f3a50009750 comp rx=0 tx=0).stop 2024-04-18T10:05:56.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.138+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/1277527338 shutdown_connections 2024-04-18T10:05:56.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.138+0000 7f3a5f2a6640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3a3c0683b0 0x7f3a3c06a870 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.138+0000 7f3a5f2a6640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a580b0e90 0x7f3a58071660 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.138+0000 7f3a5f2a6640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a580b0a10 0x7f3a581cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.138+0000 7f3a5f2a6640 1 --2- 172.21.15.62:0/1277527338 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a440051d0 0x7f3a581d09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.138+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/1277527338 >> 172.21.15.62:0/1277527338 conn(0x7f3a580b9690 msgr2=0x7f3a581d2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:56.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.142+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/1277527338 shutdown_connections 2024-04-18T10:05:56.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.142+0000 7f3a5f2a6640 1 -- 172.21.15.62:0/1277527338 wait complete. 2024-04-18T10:05:56.154 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-18T10:05:56.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.274+0000 7fc3ab479640 1 Processor -- start 2024-04-18T10:05:56.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.274+0000 7fc3ab479640 1 -- start start 2024-04-18T10:05:56.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc3a40b7f70 0x7fc3a4141900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:56.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a4141e40 0x7fc3a41525b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:56.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a40b6740 con 0x7fc3a40b6360 2024-04-18T10:05:56.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fc3a40b7720 con 0x7fc3a40b7f70 2024-04-18T10:05:56.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc3a4147e40 con 0x7fc3a4141e40 2024-04-18T10:05:56.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3a8a14640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc3a40b7f70 0x7fc3a4141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:56.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3a8a14640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc3a40b7f70 0x7fc3a4141900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40690/0 (socket says 172.21.15.62:40690) 2024-04-18T10:05:56.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3a8a14640 1 -- 172.21.15.62:0/1180517294 learned_addr learned my addr 172.21.15.62:0/1180517294 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:56.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3a9a16640 1 --2- 172.21.15.62:0/1180517294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a4141e40 0x7fc3a41525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:56.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3a8a14640 1 -- 172.21.15.62:0/1180517294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a4141e40 msgr2=0x7fc3a41525b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:56.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3a8a14640 1 --2- 172.21.15.62:0/1180517294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a4141e40 0x7fc3a41525b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3a8a14640 1 -- 172.21.15.62:0/1180517294 >> v1:172.21.15.145:6789/0 conn(0x7fc3a40b6360 legacy=0x7fc3a4148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:56.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3a8a14640 1 -- 172.21.15.62:0/1180517294 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a4148120 con 0x7fc3a40b7f70 2024-04-18T10:05:56.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3a9a16640 1 --2- 172.21.15.62:0/1180517294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a4141e40 0x7fc3a41525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:05:56.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3a8a14640 1 --2- 172.21.15.62:0/1180517294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3a40b7f70 0x7fc3a4141900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009870 tx=0x7fc3a001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b4ff502efb6b7f server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:56.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc39bfff640 1 -- 172.21.15.62:0/1180517294 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a00b0070 con 0x7fc3a40b7f70 2024-04-18T10:05:56.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc39bfff640 1 -- 172.21.15.62:0/1180517294 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a00ab600 con 0x7fc3a40b7f70 2024-04-18T10:05:56.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc39bfff640 1 -- 172.21.15.62:0/1180517294 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a00ab9d0 con 0x7fc3a40b7f70 2024-04-18T10:05:56.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 -- 172.21.15.62:0/1180517294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3a40b7f70 msgr2=0x7fc3a4141900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:56.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 --2- 172.21.15.62:0/1180517294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3a40b7f70 0x7fc3a4141900 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009870 tx=0x7fc3a001c490 comp rx=0 tx=0).stop 2024-04-18T10:05:56.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 -- 172.21.15.62:0/1180517294 shutdown_connections 2024-04-18T10:05:56.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 --2- 172.21.15.62:0/1180517294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a4141e40 0x7fc3a41525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 --2- 172.21.15.62:0/1180517294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3a40b7f70 0x7fc3a4141900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 -- 172.21.15.62:0/1180517294 >> 172.21.15.62:0/1180517294 conn(0x7fc3a40ba070 msgr2=0x7fc3a40ba480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:56.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 -- 172.21.15.62:0/1180517294 shutdown_connections 2024-04-18T10:05:56.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.278+0000 7fc3ab479640 1 -- 172.21.15.62:0/1180517294 wait complete. 2024-04-18T10:05:56.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3ab479640 1 Processor -- start 2024-04-18T10:05:56.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3ab479640 1 -- start start 2024-04-18T10:05:56.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3ab479640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3a40b6360 0x7fc3a41450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:56.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3ab479640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a40b7f70 0x7fc3a4149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3ab479640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3a4141e40 0x7fc3a41e3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3ab479640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a40b7240 con 0x7fc3a40b6360 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3ab479640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a40b73c0 con 0x7fc3a4141e40 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3ab479640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc3a4155310 con 0x7fc3a40b7f70 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3a9215640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3a40b6360 0x7fc3a41450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3a9215640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3a40b6360 0x7fc3a41450f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40692/0 (socket says 172.21.15.62:40692) 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3a9215640 1 -- 172.21.15.62:0/1623598220 learned_addr learned my addr 172.21.15.62:0/1623598220 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3a9a16640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3a4141e40 0x7fc3a41e3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3a9215640 1 -- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a40b7f70 msgr2=0x7fc3a4149470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3a8a14640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a40b7f70 0x7fc3a4149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3a9215640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a40b7f70 0x7fc3a4149470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3a9215640 1 -- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3a4141e40 msgr2=0x7fc3a41e3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:56.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3a9215640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3a4141e40 0x7fc3a41e3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3a9215640 1 -- 172.21.15.62:0/1623598220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a4144260 con 0x7fc3a40b6360 2024-04-18T10:05:56.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3a9a16640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3a4141e40 0x7fc3a41e3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:05:56.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3a9215640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3a40b6360 0x7fc3a41450f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c480 tx=0x7fc39400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:56.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc399ffb640 1 -- 172.21.15.62:0/1623598220 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394012830 con 0x7fc3a40b6360 2024-04-18T10:05:56.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc399ffb640 1 -- 172.21.15.62:0/1623598220 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc394017070 con 0x7fc3a40b6360 2024-04-18T10:05:56.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3ab479640 1 -- 172.21.15.62:0/1623598220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a4144540 con 0x7fc3a40b6360 2024-04-18T10:05:56.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc399ffb640 1 -- 172.21.15.62:0/1623598220 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394012c30 con 0x7fc3a40b6360 2024-04-18T10:05:56.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3ab479640 1 -- 172.21.15.62:0/1623598220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a41e41f0 con 0x7fc3a40b6360 2024-04-18T10:05:56.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc3ab479640 1 -- 172.21.15.62:0/1623598220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3a4069230 con 0x7fc3a40b6360 2024-04-18T10:05:56.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.282+0000 7fc399ffb640 1 -- 172.21.15.62:0/1623598220 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc39401c460 con 0x7fc3a40b6360 2024-04-18T10:05:56.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.286+0000 7fc399ffb640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc388068380 0x7fc38806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:56.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.286+0000 7fc399ffb640 1 -- 172.21.15.62:0/1623598220 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fc3940864a0 con 0x7fc3a40b6360 2024-04-18T10:05:56.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.286+0000 7fc3a8a14640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc388068380 0x7fc38806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:56.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.286+0000 7fc3a8a14640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc388068380 0x7fc38806a840 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00ad000 tx=0x7fc3a0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:56.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.290+0000 7fc399ffb640 1 -- 172.21.15.62:0/1623598220 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc394056e40 con 0x7fc3a40b6360 2024-04-18T10:05:56.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.454+0000 7fc3ab479640 1 -- 172.21.15.62:0/1623598220 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc3a40b8a20 con 0x7fc3a40b6360 2024-04-18T10:05:56.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.454+0000 7fc399ffb640 1 -- 172.21.15.62:0/1623598220 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7fc394056740 con 0x7fc3a40b6360 2024-04-18T10:05:56.454 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:56.455 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-04-18T10:05:56.455 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:05:56.455 INFO:teuthology.orchestra.run.smithi062.stdout: "poolnum": 1, 2024-04-18T10:05:56.455 INFO:teuthology.orchestra.run.smithi062.stdout: "poolname": ".mgr" 2024-04-18T10:05:56.455 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:05:56.455 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-04-18T10:05:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.458+0000 7fc3ab479640 1 -- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc388068380 msgr2=0x7fc38806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.458+0000 7fc3ab479640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc388068380 0x7fc38806a840 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00ad000 tx=0x7fc3a0008040 comp rx=0 tx=0).stop 2024-04-18T10:05:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.458+0000 7fc3ab479640 1 -- 172.21.15.62:0/1623598220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3a40b6360 msgr2=0x7fc3a41450f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.458+0000 7fc3ab479640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3a40b6360 0x7fc3a41450f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c480 tx=0x7fc39400c940 comp rx=0 tx=0).stop 2024-04-18T10:05:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.458+0000 7fc3ab479640 1 -- 172.21.15.62:0/1623598220 shutdown_connections 2024-04-18T10:05:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.458+0000 7fc3ab479640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc388068380 0x7fc38806a840 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.458+0000 7fc3ab479640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3a4141e40 0x7fc3a41e3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.458+0000 7fc3ab479640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3a40b7f70 0x7fc3a4149470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.458+0000 7fc3ab479640 1 --2- 172.21.15.62:0/1623598220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3a40b6360 0x7fc3a41450f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.458+0000 7fc3ab479640 1 -- 172.21.15.62:0/1623598220 >> 172.21.15.62:0/1623598220 conn(0x7fc3a40ba070 msgr2=0x7fc3a40b8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:56.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.458+0000 7fc3ab479640 1 -- 172.21.15.62:0/1623598220 shutdown_connections 2024-04-18T10:05:56.458 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.458+0000 7fc3ab479640 1 -- 172.21.15.62:0/1623598220 wait complete. 2024-04-18T10:05:56.474 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-18T10:05:56.474 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-18T10:05:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.598+0000 7f8b53330640 1 Processor -- start 2024-04-18T10:05:56.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.598+0000 7f8b53330640 1 -- start start 2024-04-18T10:05:56.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.598+0000 7f8b53330640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8b4c067a70 0x7f8b4c06ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.598+0000 7f8b53330640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8b4c150530 0x7f8b4c152920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.598+0000 7f8b53330640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b4c0af4e0 con 0x7f8b4c06db10 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.598+0000 7f8b53330640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f8b4c06f140 con 0x7f8b4c067a70 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b53330640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f8b4c0affc0 con 0x7f8b4c150530 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b510cc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8b4c067a70 0x7f8b4c06ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b510cc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8b4c067a70 0x7f8b4c06ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40702/0 (socket says 172.21.15.62:40702) 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b508cb640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8b4c06db10 0x7f8b4c1450c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58618/0 (socket says 172.21.15.62:58618) 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b510cc640 1 -- 172.21.15.62:0/20186091 learned_addr learned my addr 172.21.15.62:0/20186091 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b518cd640 1 --2- 172.21.15.62:0/20186091 >> v2:172.21.15.145:3300/0 conn(0x7f8b4c150530 0x7f8b4c152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b510cc640 1 -- 172.21.15.62:0/20186091 >> v2:172.21.15.145:3300/0 conn(0x7f8b4c150530 msgr2=0x7f8b4c152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b510cc640 1 --2- 172.21.15.62:0/20186091 >> v2:172.21.15.145:3300/0 conn(0x7f8b4c150530 0x7f8b4c152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b510cc640 1 -- 172.21.15.62:0/20186091 >> v1:172.21.15.62:6789/0 conn(0x7f8b4c06db10 legacy=0x7f8b4c1450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b510cc640 1 -- 172.21.15.62:0/20186091 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4c067e50 con 0x7f8b4c067a70 2024-04-18T10:05:56.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b43fff640 1 -- 172.21.15.62:0/20186091 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1762877189 0 0) 0x7f8b4c0af4e0 con 0x7f8b4c06db10 2024-04-18T10:05:56.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b510cc640 1 --2- 172.21.15.62:0/20186091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c067a70 0x7f8b4c06ec00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c06d320 tx=0x7f8b4801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48459da283cb0934 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:56.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b43fff640 1 -- 172.21.15.62:0/20186091 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b480aa670 con 0x7f8b4c067a70 2024-04-18T10:05:56.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b43fff640 1 -- 172.21.15.62:0/20186091 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b480af070 con 0x7f8b4c067a70 2024-04-18T10:05:56.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b43fff640 1 -- 172.21.15.62:0/20186091 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b480aaa30 con 0x7f8b4c067a70 2024-04-18T10:05:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b53330640 1 -- 172.21.15.62:0/20186091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c067a70 msgr2=0x7f8b4c06ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b53330640 1 --2- 172.21.15.62:0/20186091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c067a70 0x7f8b4c06ec00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c06d320 tx=0x7f8b4801c4f0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b53330640 1 -- 172.21.15.62:0/20186091 shutdown_connections 2024-04-18T10:05:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b53330640 1 --2- 172.21.15.62:0/20186091 >> v2:172.21.15.145:3300/0 conn(0x7f8b4c150530 0x7f8b4c152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b53330640 1 --2- 172.21.15.62:0/20186091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c067a70 0x7f8b4c06ec00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b53330640 1 -- 172.21.15.62:0/20186091 >> 172.21.15.62:0/20186091 conn(0x7f8b4c0b7c60 msgr2=0x7f8b4c0b8070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b53330640 1 -- 172.21.15.62:0/20186091 shutdown_connections 2024-04-18T10:05:56.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.602+0000 7f8b53330640 1 -- 172.21.15.62:0/20186091 wait complete. 2024-04-18T10:05:56.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b53330640 1 Processor -- start 2024-04-18T10:05:56.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b53330640 1 -- start start 2024-04-18T10:05:56.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b53330640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c067a70 0x7f8b4c148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:56.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b53330640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c06db10 0x7f8b4c13fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b53330640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b4c150530 0x7f8b4c13d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b53330640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4c145950 con 0x7f8b4c067a70 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b53330640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f8b4c1457d0 con 0x7f8b4c06db10 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b53330640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8b4c0b9950 con 0x7f8b4c150530 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b510cc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c067a70 0x7f8b4c148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b510cc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c067a70 0x7f8b4c148460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40718/0 (socket says 172.21.15.62:40718) 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b510cc640 1 -- 172.21.15.62:0/1306847491 learned_addr learned my addr 172.21.15.62:0/1306847491 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b508cb640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c06db10 0x7f8b4c13fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b510cc640 1 -- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b4c150530 msgr2=0x7f8b4c13d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b510cc640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b4c150530 0x7f8b4c13d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b510cc640 1 -- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c06db10 msgr2=0x7f8b4c13fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b510cc640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c06db10 0x7f8b4c13fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b510cc640 1 -- 172.21.15.62:0/1306847491 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4c141140 con 0x7f8b4c067a70 2024-04-18T10:05:56.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b510cc640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c067a70 0x7f8b4c148460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8b4801aba0 tx=0x7f8b4801ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:56.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b41ffb640 1 -- 172.21.15.62:0/1306847491 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b480b9070 con 0x7f8b4c067a70 2024-04-18T10:05:56.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b41ffb640 1 -- 172.21.15.62:0/1306847491 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b480aabe0 con 0x7f8b4c067a70 2024-04-18T10:05:56.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b53330640 1 -- 172.21.15.62:0/1306847491 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b4c0b41e0 con 0x7f8b4c067a70 2024-04-18T10:05:56.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b41ffb640 1 -- 172.21.15.62:0/1306847491 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b48004020 con 0x7f8b4c067a70 2024-04-18T10:05:56.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b53330640 1 -- 172.21.15.62:0/1306847491 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b4c13f250 con 0x7f8b4c067a70 2024-04-18T10:05:56.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b53330640 1 -- 172.21.15.62:0/1306847491 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b4c068ca0 con 0x7f8b4c067a70 2024-04-18T10:05:56.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.606+0000 7f8b41ffb640 1 -- 172.21.15.62:0/1306847491 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8b480041c0 con 0x7f8b4c067a70 2024-04-18T10:05:56.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.610+0000 7f8b41ffb640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8b300683b0 0x7f8b3006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:56.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.610+0000 7f8b41ffb640 1 -- 172.21.15.62:0/1306847491 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f8b48120ab0 con 0x7f8b4c067a70 2024-04-18T10:05:56.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.610+0000 7f8b508cb640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8b300683b0 0x7f8b3006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:56.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.610+0000 7f8b508cb640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8b300683b0 0x7f8b3006a870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c0b40f0 tx=0x7f8b3c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:56.611 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.614+0000 7f8b41ffb640 1 -- 172.21.15.62:0/1306847491 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8b48016110 con 0x7f8b4c067a70 2024-04-18T10:05:56.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:56.774+0000 7f8b53330640 1 -- 172.21.15.62:0/1306847491 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8b4c0b90f0 con 0x7f8b4c067a70 2024-04-18T10:05:57.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.186+0000 7f8b41ffb640 1 -- 172.21.15.62:0/1306847491 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f8b480a93d0 con 0x7f8b4c067a70 2024-04-18T10:05:57.186 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' created 2024-04-18T10:05:57.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.190+0000 7f8b53330640 1 -- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8b300683b0 msgr2=0x7f8b3006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:57.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.190+0000 7f8b53330640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8b300683b0 0x7f8b3006a870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c0b40f0 tx=0x7f8b3c009040 comp rx=0 tx=0).stop 2024-04-18T10:05:57.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.190+0000 7f8b53330640 1 -- 172.21.15.62:0/1306847491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c067a70 msgr2=0x7f8b4c148460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:57.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.190+0000 7f8b53330640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c067a70 0x7f8b4c148460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8b4801aba0 tx=0x7f8b4801ad30 comp rx=0 tx=0).stop 2024-04-18T10:05:57.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.190+0000 7f8b53330640 1 -- 172.21.15.62:0/1306847491 shutdown_connections 2024-04-18T10:05:57.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.190+0000 7f8b53330640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8b300683b0 0x7f8b3006a870 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:57.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.190+0000 7f8b53330640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b4c150530 0x7f8b4c13d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:57.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.190+0000 7f8b53330640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c06db10 0x7f8b4c13fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:57.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.190+0000 7f8b53330640 1 --2- 172.21.15.62:0/1306847491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8b4c067a70 0x7f8b4c148460 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:57.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.190+0000 7f8b53330640 1 -- 172.21.15.62:0/1306847491 >> 172.21.15.62:0/1306847491 conn(0x7f8b4c0b7c60 msgr2=0x7f8b4c13e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:57.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.190+0000 7f8b53330640 1 -- 172.21.15.62:0/1306847491 shutdown_connections 2024-04-18T10:05:57.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.190+0000 7f8b53330640 1 -- 172.21.15.62:0/1306847491 wait complete. 2024-04-18T10:05:57.219 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-18T10:05:57.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe16b60640 1 Processor -- start 2024-04-18T10:05:57.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe16b60640 1 -- start start 2024-04-18T10:05:57.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe16b60640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe100b6980 0x7efe1009d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:57.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe16b60640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efe1006d210 0x7efe101478b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:57.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe16b60640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe100b6d60 con 0x7efe100b5e90 2024-04-18T10:05:57.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe16b60640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7efe1009dda0 con 0x7efe1006d210 2024-04-18T10:05:57.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe16b60640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7efe1009c9a0 con 0x7efe100b6980 2024-04-18T10:05:57.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe150fd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7efe100b5e90 0x7efe10147df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58628/0 (socket says 172.21.15.62:58628) 2024-04-18T10:05:57.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe150fd640 1 -- 172.21.15.62:0/127261158 learned_addr learned my addr 172.21.15.62:0/127261158 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:05:57.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 30287585 0 0) 0x7efe100b6d60 con 0x7efe100b5e90 2024-04-18T10:05:57.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe0ffff640 1 --2- 172.21.15.62:0/127261158 >> v2:172.21.15.145:3300/0 conn(0x7efe1006d210 0x7efe101478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:57.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe00003530 con 0x7efe100b5e90 2024-04-18T10:05:57.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe148fc640 1 --2- 172.21.15.62:0/127261158 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe100b6980 0x7efe1009d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:57.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4031206335 0 0) 0x7efe00003530 con 0x7efe100b5e90 2024-04-18T10:05:57.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe100b6980 msgr2=0x7efe1009d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:57.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.354+0000 7efe0f7fe640 1 --2- 172.21.15.62:0/127261158 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe100b6980 0x7efe1009d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:57.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 >> v2:172.21.15.145:3300/0 conn(0x7efe1006d210 msgr2=0x7efe101478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:57.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 --2- 172.21.15.62:0/127261158 >> v2:172.21.15.145:3300/0 conn(0x7efe1006d210 0x7efe101478b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:57.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1009d750 con 0x7efe100b5e90 2024-04-18T10:05:57.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7efe08001a00 con 0x7efe100b5e90 2024-04-18T10:05:57.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 >> v1:172.21.15.62:6789/0 conn(0x7efe100b5e90 legacy=0x7efe10147df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 --2- 172.21.15.62:0/127261158 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe100b6980 0x7efe000039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 --2- 172.21.15.62:0/127261158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe00003f40 0x7efe00004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 --2- 172.21.15.62:0/127261158 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe00004d50 0x7efe000051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efe1009dda0 con 0x7efe00003f40 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7efe1009c9a0 con 0x7efe00004d50 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7efe0000bad0 con 0x7efe100b6980 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0ffff640 1 --2- 172.21.15.62:0/127261158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe00003f40 0x7efe00004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe148fc640 1 --2- 172.21.15.62:0/127261158 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe100b6980 0x7efe000039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe150fd640 1 --2- 172.21.15.62:0/127261158 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe00004d50 0x7efe000051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0ffff640 1 -- 172.21.15.62:0/127261158 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe100b6980 msgr2=0x7efe000039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0ffff640 1 --2- 172.21.15.62:0/127261158 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe100b6980 0x7efe000039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0ffff640 1 -- 172.21.15.62:0/127261158 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe00004d50 msgr2=0x7efe000051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:57.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0ffff640 1 --2- 172.21.15.62:0/127261158 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe00004d50 0x7efe000051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:57.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0ffff640 1 -- 172.21.15.62:0/127261158 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe00003530 con 0x7efe00003f40 2024-04-18T10:05:57.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0ffff640 1 --2- 172.21.15.62:0/127261158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe00003f40 0x7efe00004810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efdfc009b10 tx=0x7efdfc01cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1236a64f3d1934 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:57.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc0b2070 con 0x7efe00003f40 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdfc0a5550 con 0x7efe00003f40 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe0f7fe640 1 -- 172.21.15.62:0/127261158 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc0096b0 con 0x7efe00003f40 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe16b60640 1 -- 172.21.15.62:0/127261158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe00003f40 msgr2=0x7efe00004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe16b60640 1 --2- 172.21.15.62:0/127261158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe00003f40 0x7efe00004810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efdfc009b10 tx=0x7efdfc01cab0 comp rx=0 tx=0).stop 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe150fd640 1 -- 172.21.15.62:0/127261158 reap_dead start 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe16b60640 1 -- 172.21.15.62:0/127261158 shutdown_connections 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe16b60640 1 -- 172.21.15.62:0/127261158 >> 172.21.15.62:0/127261158 conn(0x7efe100b9390 msgr2=0x7efe100b97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe16b60640 1 -- 172.21.15.62:0/127261158 shutdown_connections 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.358+0000 7efe16b60640 1 -- 172.21.15.62:0/127261158 wait complete. 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe16b60640 1 Processor -- start 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe16b60640 1 -- start start 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe16b60640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe00003f40 0x7efe10140310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:57.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe16b60640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe1013e240 0x7efe10143c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe16b60640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe101e3e30 0x7efe101e42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe16b60640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efe10148680 con 0x7efe101e3e30 2024-04-18T10:05:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe16b60640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7efe10148500 con 0x7efe1013e240 2024-04-18T10:05:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe16b60640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7efe1009e210 con 0x7efe00003f40 2024-04-18T10:05:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe150fd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe101e3e30 0x7efe101e42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe150fd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe101e3e30 0x7efe101e42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40744/0 (socket says 172.21.15.62:40744) 2024-04-18T10:05:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe150fd640 1 -- 172.21.15.62:0/1583661407 learned_addr learned my addr 172.21.15.62:0/1583661407 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe148fc640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe00003f40 0x7efe10140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe0ffff640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe1013e240 0x7efe10143c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe150fd640 1 -- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe00003f40 msgr2=0x7efe10140310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe150fd640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe00003f40 0x7efe10140310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:57.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe150fd640 1 -- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe1013e240 msgr2=0x7efe10143c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:57.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe150fd640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe1013e240 0x7efe10143c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:57.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe150fd640 1 -- 172.21.15.62:0/1583661407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe100b0aa0 con 0x7efe101e3e30 2024-04-18T10:05:57.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe150fd640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe101e3e30 0x7efe101e42b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efe08002240 tx=0x7efe0800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:57.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe0d7fa640 1 -- 172.21.15.62:0/1583661407 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe08007ae0 con 0x7efe101e3e30 2024-04-18T10:05:57.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe0d7fa640 1 -- 172.21.15.62:0/1583661407 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe08018070 con 0x7efe101e3e30 2024-04-18T10:05:57.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe0d7fa640 1 -- 172.21.15.62:0/1583661407 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe08013420 con 0x7efe101e3e30 2024-04-18T10:05:57.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe16b60640 1 -- 172.21.15.62:0/1583661407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe1013fdf0 con 0x7efe101e3e30 2024-04-18T10:05:57.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe16b60640 1 -- 172.21.15.62:0/1583661407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe10073730 con 0x7efe101e3e30 2024-04-18T10:05:57.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe16b60640 1 -- 172.21.15.62:0/1583661407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe10068b90 con 0x7efe101e3e30 2024-04-18T10:05:57.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.362+0000 7efe0d7fa640 1 -- 172.21.15.62:0/1583661407 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efe080135c0 con 0x7efe101e3e30 2024-04-18T10:05:57.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.366+0000 7efe0d7fa640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efdf4068380 0x7efdf406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:57.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.366+0000 7efe0d7fa640 1 -- 172.21.15.62:0/1583661407 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7efe08087570 con 0x7efe101e3e30 2024-04-18T10:05:57.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.366+0000 7efe148fc640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efdf4068380 0x7efdf406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:57.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.366+0000 7efe148fc640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efdf4068380 0x7efdf406a840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efe0400b8d0 tx=0x7efe04008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:57.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.370+0000 7efe0d7fa640 1 -- 172.21.15.62:0/1583661407 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efe08057e80 con 0x7efe101e3e30 2024-04-18T10:05:57.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:57.534+0000 7efe16b60640 1 -- 172.21.15.62:0/1583661407 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7efe1013ffc0 con 0x7efe101e3e30 2024-04-18T10:05:58.197 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.198+0000 7efe0d7fa640 1 -- 172.21.15.62:0/1583661407 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7efe08057780 con 0x7efe101e3e30 2024-04-18T10:05:58.197 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' created 2024-04-18T10:05:58.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.202+0000 7efe16b60640 1 -- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efdf4068380 msgr2=0x7efdf406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:58.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.202+0000 7efe16b60640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efdf4068380 0x7efdf406a840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efe0400b8d0 tx=0x7efe04008040 comp rx=0 tx=0).stop 2024-04-18T10:05:58.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.202+0000 7efe16b60640 1 -- 172.21.15.62:0/1583661407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe101e3e30 msgr2=0x7efe101e42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:58.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.202+0000 7efe16b60640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe101e3e30 0x7efe101e42b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7efe08002240 tx=0x7efe0800e530 comp rx=0 tx=0).stop 2024-04-18T10:05:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.202+0000 7efe16b60640 1 -- 172.21.15.62:0/1583661407 shutdown_connections 2024-04-18T10:05:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.202+0000 7efe16b60640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efdf4068380 0x7efdf406a840 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.202+0000 7efe16b60640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efe101e3e30 0x7efe101e42b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.202+0000 7efe16b60640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe1013e240 0x7efe10143c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.202+0000 7efe16b60640 1 --2- 172.21.15.62:0/1583661407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe00003f40 0x7efe10140310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.202+0000 7efe16b60640 1 -- 172.21.15.62:0/1583661407 >> 172.21.15.62:0/1583661407 conn(0x7efe100b9390 msgr2=0x7efe100b7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.202+0000 7efe16b60640 1 -- 172.21.15.62:0/1583661407 shutdown_connections 2024-04-18T10:05:58.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.202+0000 7efe16b60640 1 -- 172.21.15.62:0/1583661407 wait complete. 2024-04-18T10:05:58.234 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-18T10:05:58.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeb056c640 1 Processor -- start 2024-04-18T10:05:58.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeb056c640 1 -- start start 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeb056c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80b01a0 0x7fdea806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeb056c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdea80ae4e0 0x7fdea813e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeb056c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdea80b0580 con 0x7fdea80737c0 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeb056c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fdea80b59d0 con 0x7fdea80ae4e0 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeb056c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fdea80ae970 con 0x7fdea80b01a0 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeaeb09640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fdea80737c0 0x7fdea8148100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58642/0 (socket says 172.21.15.62:58642) 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeaeb09640 1 -- 172.21.15.62:0/250419251 learned_addr learned my addr 172.21.15.62:0/250419251 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeadb07640 1 --2- 172.21.15.62:0/250419251 >> v2:172.21.15.145:3300/0 conn(0x7fdea80ae4e0 0x7fdea813e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeae308640 1 --2- 172.21.15.62:0/250419251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80b01a0 0x7fdea806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 323538778 0 0) 0x7fdea80b0580 con 0x7fdea80737c0 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde94003530 con 0x7fdea80737c0 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2347093901 0 0) 0x7fde94003530 con 0x7fdea80737c0 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80b01a0 msgr2=0x7fdea806d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:58.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 --2- 172.21.15.62:0/250419251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80b01a0 0x7fdea806d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 >> v2:172.21.15.145:3300/0 conn(0x7fdea80ae4e0 msgr2=0x7fdea813e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 --2- 172.21.15.62:0/250419251 >> v2:172.21.15.145:3300/0 conn(0x7fdea80ae4e0 0x7fdea813e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea80b53a0 con 0x7fdea80737c0 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fdea0001a00 con 0x7fdea80737c0 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 >> v1:172.21.15.62:6789/0 conn(0x7fdea80737c0 legacy=0x7fdea8148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 --2- 172.21.15.62:0/250419251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80b01a0 0x7fde940039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 --2- 172.21.15.62:0/250419251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde94003f40 0x7fde94004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 --2- 172.21.15.62:0/250419251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde94004d50 0x7fde940051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdea80b59d0 con 0x7fde94004d50 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fdea80ae970 con 0x7fde94003f40 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fde9400bab0 con 0x7fdea80b01a0 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeaeb09640 1 --2- 172.21.15.62:0/250419251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde94004d50 0x7fde940051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeae308640 1 --2- 172.21.15.62:0/250419251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80b01a0 0x7fde940039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeadb07640 1 --2- 172.21.15.62:0/250419251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde94003f40 0x7fde94004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeaeb09640 1 -- 172.21.15.62:0/250419251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80b01a0 msgr2=0x7fde940039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:58.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeaeb09640 1 --2- 172.21.15.62:0/250419251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80b01a0 0x7fde940039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:58.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeaeb09640 1 -- 172.21.15.62:0/250419251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde94003f40 msgr2=0x7fde94004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:58.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeaeb09640 1 --2- 172.21.15.62:0/250419251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fde94003f40 0x7fde94004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:58.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.370+0000 7fdeaeb09640 1 -- 172.21.15.62:0/250419251 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde94003530 con 0x7fde94004d50 2024-04-18T10:05:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdeaeb09640 1 --2- 172.21.15.62:0/250419251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde94004d50 0x7fde940051d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdea000a870 tx=0x7fdea001d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c73b0ca94e8f80b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea000a4c0 con 0x7fde94004d50 2024-04-18T10:05:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdea00a5e50 con 0x7fde94004d50 2024-04-18T10:05:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdead306640 1 -- 172.21.15.62:0/250419251 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea00ad870 con 0x7fde94004d50 2024-04-18T10:05:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdeb056c640 1 -- 172.21.15.62:0/250419251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde94004d50 msgr2=0x7fde940051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdeb056c640 1 --2- 172.21.15.62:0/250419251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde94004d50 0x7fde940051d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fdea000a870 tx=0x7fdea001d3a0 comp rx=0 tx=0).stop 2024-04-18T10:05:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdeaeb09640 1 -- 172.21.15.62:0/250419251 reap_dead start 2024-04-18T10:05:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdeb056c640 1 -- 172.21.15.62:0/250419251 shutdown_connections 2024-04-18T10:05:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdeb056c640 1 -- 172.21.15.62:0/250419251 >> 172.21.15.62:0/250419251 conn(0x7fdea80b9690 msgr2=0x7fdea80b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdeb056c640 1 -- 172.21.15.62:0/250419251 shutdown_connections 2024-04-18T10:05:58.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdeb056c640 1 -- 172.21.15.62:0/250419251 wait complete. 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdeb056c640 1 Processor -- start 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.374+0000 7fdeb056c640 1 -- start start 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeb056c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde94004d50 0x7fdea8070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeb056c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80721f0 0x7fdea809c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeb056c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdea813b8c0 0x7fdea813bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeb056c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdea806d6b0 con 0x7fde94004d50 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeb056c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fdea806e470 con 0x7fdea813b8c0 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeb056c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fdea806d830 con 0x7fdea80721f0 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeae308640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde94004d50 0x7fdea8070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeae308640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde94004d50 0x7fdea8070940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40766/0 (socket says 172.21.15.62:40766) 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeae308640 1 -- 172.21.15.62:0/2175024513 learned_addr learned my addr 172.21.15.62:0/2175024513 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeaeb09640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdea813b8c0 0x7fdea813bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeadb07640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80721f0 0x7fdea809c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeae308640 1 -- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80721f0 msgr2=0x7fdea809c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:58.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeae308640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80721f0 0x7fdea809c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeae308640 1 -- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdea813b8c0 msgr2=0x7fdea813bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeae308640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdea813b8c0 0x7fdea813bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeae308640 1 -- 172.21.15.62:0/2175024513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea809c8a0 con 0x7fde94004d50 2024-04-18T10:05:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeae308640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde94004d50 0x7fdea8070940 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdea40115e0 tx=0x7fdea4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fde9effd640 1 -- 172.21.15.62:0/2175024513 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea400f8f0 con 0x7fde94004d50 2024-04-18T10:05:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fde9effd640 1 -- 172.21.15.62:0/2175024513 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdea400fa90 con 0x7fde94004d50 2024-04-18T10:05:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fde9effd640 1 -- 172.21.15.62:0/2175024513 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdea400fe60 con 0x7fde94004d50 2024-04-18T10:05:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeb056c640 1 -- 172.21.15.62:0/2175024513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdea813c280 con 0x7fde94004d50 2024-04-18T10:05:58.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeb056c640 1 -- 172.21.15.62:0/2175024513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdea80719f0 con 0x7fde94004d50 2024-04-18T10:05:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fdeb056c640 1 -- 172.21.15.62:0/2175024513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdea8068c00 con 0x7fde94004d50 2024-04-18T10:05:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fde9effd640 1 -- 172.21.15.62:0/2175024513 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdea4015460 con 0x7fde94004d50 2024-04-18T10:05:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fde9effd640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fde8c068380 0x7fde8c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.378+0000 7fde9effd640 1 -- 172.21.15.62:0/2175024513 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fdea40872b0 con 0x7fde94004d50 2024-04-18T10:05:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.382+0000 7fdeadb07640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fde8c068380 0x7fde8c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:58.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.382+0000 7fdeadb07640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fde8c068380 0x7fde8c06a840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdea813c750 tx=0x7fde98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:58.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.386+0000 7fde9effd640 1 -- 172.21.15.62:0/2175024513 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdea4019030 con 0x7fde94004d50 2024-04-18T10:05:58.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:58.586+0000 7fdeb056c640 1 -- 172.21.15.62:0/2175024513 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fdea8073270 con 0x7fde94004d50 2024-04-18T10:05:58.826 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-18T10:05:58.829 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-18T10:05:59.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.222+0000 7fde9effd640 1 -- 172.21.15.62:0/2175024513 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fdea4057960 con 0x7fde94004d50 2024-04-18T10:05:59.224 INFO:teuthology.orchestra.run.smithi062.stderr:new fs with metadata pool 2 and data pool 3 2024-04-18T10:05:59.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.234+0000 7fdeb056c640 1 -- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fde8c068380 msgr2=0x7fde8c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:59.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.234+0000 7fdeb056c640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fde8c068380 0x7fde8c06a840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fdea813c750 tx=0x7fde98008040 comp rx=0 tx=0).stop 2024-04-18T10:05:59.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.234+0000 7fdeb056c640 1 -- 172.21.15.62:0/2175024513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde94004d50 msgr2=0x7fdea8070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:59.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.234+0000 7fdeb056c640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde94004d50 0x7fdea8070940 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdea40115e0 tx=0x7fdea4011aa0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.234+0000 7fdeb056c640 1 -- 172.21.15.62:0/2175024513 shutdown_connections 2024-04-18T10:05:59.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.234+0000 7fdeb056c640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fde8c068380 0x7fde8c06a840 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.234+0000 7fdeb056c640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdea813b8c0 0x7fdea813bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.233 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.234+0000 7fdeb056c640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdea80721f0 0x7fdea809c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.234+0000 7fdeb056c640 1 --2- 172.21.15.62:0/2175024513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fde94004d50 0x7fdea8070940 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.234+0000 7fdeb056c640 1 -- 172.21.15.62:0/2175024513 >> 172.21.15.62:0/2175024513 conn(0x7fdea80b9690 msgr2=0x7fdea80b4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:59.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.234+0000 7fdeb056c640 1 -- 172.21.15.62:0/2175024513 shutdown_connections 2024-04-18T10:05:59.234 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.234+0000 7fdeb056c640 1 -- 172.21.15.62:0/2175024513 wait complete. 2024-04-18T10:05:59.261 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:05:59.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.378+0000 7ffab1a9d640 1 Processor -- start 2024-04-18T10:05:59.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.378+0000 7ffab1a9d640 1 -- start start 2024-04-18T10:05:59.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffab1a9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ffaac070260 0x7ffaac148600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffab1a9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0735d0 0x7ffaac14b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffab1a9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffaac148b40 0x7ffaac14b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffab1a9d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7ffaac148170 con 0x7ffaac148b40 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffab1a9d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7ffaac0ba360 con 0x7ffaac070260 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffab1a9d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7ffaac070640 con 0x7ffaac0735d0 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaabfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffaac148b40 0x7ffaac14b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaabfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffaac148b40 0x7ffaac14b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40778/0 (socket says 172.21.15.62:40778) 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaabfff640 1 -- 172.21.15.62:0/3014301750 learned_addr learned my addr 172.21.15.62:0/3014301750 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaab7fe640 1 --2- 172.21.15.62:0/3014301750 >> v2:172.21.15.145:3300/0 conn(0x7ffaac070260 0x7ffaac148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaaaffd640 1 --2- 172.21.15.62:0/3014301750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0735d0 0x7ffaac14b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaabfff640 1 -- 172.21.15.62:0/3014301750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0735d0 msgr2=0x7ffaac14b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaabfff640 1 --2- 172.21.15.62:0/3014301750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0735d0 0x7ffaac14b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaabfff640 1 -- 172.21.15.62:0/3014301750 >> v2:172.21.15.145:3300/0 conn(0x7ffaac070260 msgr2=0x7ffaac148600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaabfff640 1 --2- 172.21.15.62:0/3014301750 >> v2:172.21.15.145:3300/0 conn(0x7ffaac070260 0x7ffaac148600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaabfff640 1 -- 172.21.15.62:0/3014301750 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac06fc00 con 0x7ffaac148b40 2024-04-18T10:05:59.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaabfff640 1 --2- 172.21.15.62:0/3014301750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaac148b40 0x7ffaac14b9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0bc7f0 tx=0x7ffa9401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf62f09d93f2a08 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:59.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaaa7fc640 1 -- 172.21.15.62:0/3014301750 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa940b0070 con 0x7ffaac148b40 2024-04-18T10:05:59.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaaa7fc640 1 -- 172.21.15.62:0/3014301750 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa940ab600 con 0x7ffaac148b40 2024-04-18T10:05:59.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.382+0000 7ffaaa7fc640 1 -- 172.21.15.62:0/3014301750 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa940ab9d0 con 0x7ffaac148b40 2024-04-18T10:05:59.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 -- 172.21.15.62:0/3014301750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaac148b40 msgr2=0x7ffaac14b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:59.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 --2- 172.21.15.62:0/3014301750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaac148b40 0x7ffaac14b9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffaac0bc7f0 tx=0x7ffa9401c520 comp rx=0 tx=0).stop 2024-04-18T10:05:59.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 -- 172.21.15.62:0/3014301750 shutdown_connections 2024-04-18T10:05:59.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 --2- 172.21.15.62:0/3014301750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaac148b40 0x7ffaac14b9a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 --2- 172.21.15.62:0/3014301750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0735d0 0x7ffaac14b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 --2- 172.21.15.62:0/3014301750 >> v2:172.21.15.145:3300/0 conn(0x7ffaac070260 0x7ffaac148600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 -- 172.21.15.62:0/3014301750 >> 172.21.15.62:0/3014301750 conn(0x7ffaac0a0f00 msgr2=0x7ffaac09aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:59.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 -- 172.21.15.62:0/3014301750 shutdown_connections 2024-04-18T10:05:59.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 -- 172.21.15.62:0/3014301750 wait complete. 2024-04-18T10:05:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 Processor -- start 2024-04-18T10:05:59.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 -- start start 2024-04-18T10:05:59.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaac070260 0x7ffaac09d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0735d0 0x7ffaac1e0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffaac148b40 0x7ffaac144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac1422e0 con 0x7ffaac070260 2024-04-18T10:05:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac14c2f0 con 0x7ffaac148b40 2024-04-18T10:05:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7ffaac142460 con 0x7ffaac0735d0 2024-04-18T10:05:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaab7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaac070260 0x7ffaac09d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaab7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaac070260 0x7ffaac09d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40790/0 (socket says 172.21.15.62:40790) 2024-04-18T10:05:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaab7fe640 1 -- 172.21.15.62:0/2055055782 learned_addr learned my addr 172.21.15.62:0/2055055782 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaaaffd640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0735d0 0x7ffaac1e0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaabfff640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffaac148b40 0x7ffaac144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaab7fe640 1 -- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0735d0 msgr2=0x7ffaac1e0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaab7fe640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0735d0 0x7ffaac1e0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaab7fe640 1 -- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffaac148b40 msgr2=0x7ffaac144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:59.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaab7fe640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffaac148b40 0x7ffaac144320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaab7fe640 1 -- 172.21.15.62:0/2055055782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac1e17c0 con 0x7ffaac070260 2024-04-18T10:05:59.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaab7fe640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaac070260 0x7ffaac09d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4007f20 tx=0x7ffaa400e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:59.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaa3fff640 1 -- 172.21.15.62:0/2055055782 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4014620 con 0x7ffaac070260 2024-04-18T10:05:59.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaa3fff640 1 -- 172.21.15.62:0/2055055782 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaa4019070 con 0x7ffaac070260 2024-04-18T10:05:59.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffaa3fff640 1 -- 172.21.15.62:0/2055055782 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4014a00 con 0x7ffaac070260 2024-04-18T10:05:59.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.386+0000 7ffab1a9d640 1 -- 172.21.15.62:0/2055055782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaac1e43e0 con 0x7ffaac070260 2024-04-18T10:05:59.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.390+0000 7ffab1a9d640 1 -- 172.21.15.62:0/2055055782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaac0b6280 con 0x7ffaac070260 2024-04-18T10:05:59.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.390+0000 7ffab1a9d640 1 -- 172.21.15.62:0/2055055782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaac069230 con 0x7ffaac070260 2024-04-18T10:05:59.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.390+0000 7ffaa3fff640 1 -- 172.21.15.62:0/2055055782 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffaa4014ba0 con 0x7ffaac070260 2024-04-18T10:05:59.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.390+0000 7ffaa3fff640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ffa8c068380 0x7ffa8c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.390+0000 7ffaa3fff640 1 -- 172.21.15.62:0/2055055782 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ffaa40871e0 con 0x7ffaac070260 2024-04-18T10:05:59.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.390+0000 7ffaaaffd640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ffa8c068380 0x7ffa8c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.390+0000 7ffaaaffd640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ffa8c068380 0x7ffa8c06a840 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffaac145a40 tx=0x7ffa9c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:59.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.394+0000 7ffaa3fff640 1 -- 172.21.15.62:0/2055055782 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ffaa4057830 con 0x7ffaac070260 2024-04-18T10:05:59.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.562+0000 7ffab1a9d640 1 -- 172.21.15.62:0/2055055782 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffaac0754d0 con 0x7ffaac070260 2024-04-18T10:05:59.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffaa3fff640 1 -- 172.21.15.62:0/2055055782 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16606 (secure 0 0 0) 0x7ffaa4057130 con 0x7ffaac070260 2024-04-18T10:05:59.563 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:59.564 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":20,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:05:58.588136+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:05:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffab1a9d640 1 -- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ffa8c068380 msgr2=0x7ffa8c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffab1a9d640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ffa8c068380 0x7ffa8c06a840 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffaac145a40 tx=0x7ffa9c008040 comp rx=0 tx=0).stop 2024-04-18T10:05:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffab1a9d640 1 -- 172.21.15.62:0/2055055782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaac070260 msgr2=0x7ffaac09d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffab1a9d640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaac070260 0x7ffaac09d390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4007f20 tx=0x7ffaa400e570 comp rx=0 tx=0).stop 2024-04-18T10:05:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffab1a9d640 1 -- 172.21.15.62:0/2055055782 shutdown_connections 2024-04-18T10:05:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffab1a9d640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ffa8c068380 0x7ffa8c06a840 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffab1a9d640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffaac148b40 0x7ffaac144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffab1a9d640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0735d0 0x7ffaac1e0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffab1a9d640 1 --2- 172.21.15.62:0/2055055782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffaac070260 0x7ffaac09d390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffab1a9d640 1 -- 172.21.15.62:0/2055055782 >> 172.21.15.62:0/2055055782 conn(0x7ffaac0a0f00 msgr2=0x7ffaac06d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffab1a9d640 1 -- 172.21.15.62:0/2055055782 shutdown_connections 2024-04-18T10:05:59.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.566+0000 7ffab1a9d640 1 -- 172.21.15.62:0/2055055782 wait complete. 2024-04-18T10:05:59.583 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:05:59.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d666a9640 1 Processor -- start 2024-04-18T10:05:59.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d666a9640 1 -- start start 2024-04-18T10:05:59.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d666a9640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5d600b8db0 0x7f5d600b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d666a9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d60147f60 0x7f5d6014f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d666a9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d600b6280 con 0x7f5d6013ebd0 2024-04-18T10:05:59.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d666a9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f5d600b68b0 con 0x7f5d60147f60 2024-04-18T10:05:59.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d666a9640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f5d600b6d40 con 0x7f5d600b8db0 2024-04-18T10:05:59.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d64c46640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d60147f60 0x7f5d6014f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d64c46640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5d60147f60 0x7f5d6014f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40796/0 (socket says 172.21.15.62:40796) 2024-04-18T10:05:59.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d64c46640 1 -- 172.21.15.62:0/4062473741 learned_addr learned my addr 172.21.15.62:0/4062473741 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d577fe640 1 --2- 172.21.15.62:0/4062473741 >> v2:172.21.15.145:3300/0 conn(0x7f5d600b8db0 0x7f5d600b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d64c46640 1 -- 172.21.15.62:0/4062473741 >> v2:172.21.15.145:3300/0 conn(0x7f5d600b8db0 msgr2=0x7f5d600b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d64c46640 1 --2- 172.21.15.62:0/4062473741 >> v2:172.21.15.145:3300/0 conn(0x7f5d600b8db0 0x7f5d600b82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d64c46640 1 -- 172.21.15.62:0/4062473741 >> v1:172.21.15.145:6789/0 conn(0x7f5d6013ebd0 legacy=0x7f5d600b9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:05:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d64c46640 1 -- 172.21.15.62:0/4062473741 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d600b6540 con 0x7f5d60147f60 2024-04-18T10:05:59.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d577fe640 1 --2- 172.21.15.62:0/4062473741 >> v2:172.21.15.145:3300/0 conn(0x7f5d600b8db0 0x7f5d600b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:05:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.706+0000 7f5d64c46640 1 --2- 172.21.15.62:0/4062473741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d60147f60 0x7f5d6014f2d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5d600b87e0 tx=0x7f5d5801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=996ce52e12c33237 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d5f7fe640 1 -- 172.21.15.62:0/4062473741 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d580b0070 con 0x7f5d60147f60 2024-04-18T10:05:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d5f7fe640 1 -- 172.21.15.62:0/4062473741 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d580ab600 con 0x7f5d60147f60 2024-04-18T10:05:59.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d5f7fe640 1 -- 172.21.15.62:0/4062473741 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d580ab9d0 con 0x7f5d60147f60 2024-04-18T10:05:59.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 -- 172.21.15.62:0/4062473741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d60147f60 msgr2=0x7f5d6014f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:59.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 --2- 172.21.15.62:0/4062473741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d60147f60 0x7f5d6014f2d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5d600b87e0 tx=0x7f5d5801c450 comp rx=0 tx=0).stop 2024-04-18T10:05:59.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 -- 172.21.15.62:0/4062473741 shutdown_connections 2024-04-18T10:05:59.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 --2- 172.21.15.62:0/4062473741 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d60147f60 0x7f5d6014f2d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 --2- 172.21.15.62:0/4062473741 >> v2:172.21.15.145:3300/0 conn(0x7f5d600b8db0 0x7f5d600b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 -- 172.21.15.62:0/4062473741 >> 172.21.15.62:0/4062473741 conn(0x7f5d6009c250 msgr2=0x7f5d6009c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:59.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 -- 172.21.15.62:0/4062473741 shutdown_connections 2024-04-18T10:05:59.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 -- 172.21.15.62:0/4062473741 wait complete. 2024-04-18T10:05:59.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 Processor -- start 2024-04-18T10:05:59.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 -- start start 2024-04-18T10:05:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d600b8db0 0x7f5d600b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d6013ebd0 0x7f5d601e1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d60147f60 0x7f5d601e2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5d601523c0 con 0x7f5d6013ebd0 2024-04-18T10:05:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f5d60152240 con 0x7f5d600b8db0 2024-04-18T10:05:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f5d60152540 con 0x7f5d60147f60 2024-04-18T10:05:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d577fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d6013ebd0 0x7f5d601e1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d577fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d6013ebd0 0x7f5d601e1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40800/0 (socket says 172.21.15.62:40800) 2024-04-18T10:05:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d577fe640 1 -- 172.21.15.62:0/3672950817 learned_addr learned my addr 172.21.15.62:0/3672950817 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:05:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d5ffff640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d600b8db0 0x7f5d600b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d64c46640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d60147f60 0x7f5d601e2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d577fe640 1 -- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d60147f60 msgr2=0x7f5d601e2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d577fe640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d60147f60 0x7f5d601e2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d577fe640 1 -- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d600b8db0 msgr2=0x7f5d600b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d577fe640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d600b8db0 0x7f5d600b43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d577fe640 1 -- 172.21.15.62:0/3672950817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d60140270 con 0x7f5d6013ebd0 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d5ffff640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d600b8db0 0x7f5d600b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d64c46640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d60147f60 0x7f5d601e2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d577fe640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d6013ebd0 0x7f5d601e1c40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d4800c7f0 tx=0x7f5d4800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d5d7fa640 1 -- 172.21.15.62:0/3672950817 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d48007a30 con 0x7f5d6013ebd0 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d5d7fa640 1 -- 172.21.15.62:0/3672950817 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d48007bd0 con 0x7f5d6013ebd0 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 -- 172.21.15.62:0/3672950817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d60140550 con 0x7f5d6013ebd0 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d5d7fa640 1 -- 172.21.15.62:0/3672950817 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d480113f0 con 0x7f5d6013ebd0 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.710+0000 7f5d666a9640 1 -- 172.21.15.62:0/3672950817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d6006bfe0 con 0x7f5d6013ebd0 2024-04-18T10:05:59.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.714+0000 7f5d666a9640 1 -- 172.21.15.62:0/3672950817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d60069230 con 0x7f5d6013ebd0 2024-04-18T10:05:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.714+0000 7f5d5d7fa640 1 -- 172.21.15.62:0/3672950817 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5d48011590 con 0x7f5d6013ebd0 2024-04-18T10:05:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.714+0000 7f5d5d7fa640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5d40068380 0x7f5d4006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:05:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.714+0000 7f5d5d7fa640 1 -- 172.21.15.62:0/3672950817 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5d48086b60 con 0x7f5d6013ebd0 2024-04-18T10:05:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.714+0000 7f5d5ffff640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5d40068380 0x7f5d4006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:05:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.714+0000 7f5d5ffff640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5d40068380 0x7f5d4006a840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d50007fd0 tx=0x7f5d50009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:05:59.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.718+0000 7f5d5d7fa640 1 -- 172.21.15.62:0/3672950817 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5d48015020 con 0x7f5d6013ebd0 2024-04-18T10:05:59.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.878+0000 7f5d666a9640 1 -- 172.21.15.62:0/3672950817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d6014f130 con 0x7f5d6013ebd0 2024-04-18T10:05:59.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d5d7fa640 1 -- 172.21.15.62:0/3672950817 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16606 (secure 0 0 0) 0x7f5d480571b0 con 0x7f5d6013ebd0 2024-04-18T10:05:59.880 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:05:59.880 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":20,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:05:58.588136+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:05:59.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d666a9640 1 -- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5d40068380 msgr2=0x7f5d4006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:59.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d666a9640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5d40068380 0x7f5d4006a840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5d50007fd0 tx=0x7f5d50009040 comp rx=0 tx=0).stop 2024-04-18T10:05:59.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d666a9640 1 -- 172.21.15.62:0/3672950817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d6013ebd0 msgr2=0x7f5d601e1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:05:59.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d666a9640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d6013ebd0 0x7f5d601e1c40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d4800c7f0 tx=0x7f5d4800ccb0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d666a9640 1 -- 172.21.15.62:0/3672950817 shutdown_connections 2024-04-18T10:05:59.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d666a9640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5d40068380 0x7f5d4006a840 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d666a9640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d60147f60 0x7f5d601e2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d666a9640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5d6013ebd0 0x7f5d601e1c40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d666a9640 1 --2- 172.21.15.62:0/3672950817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d600b8db0 0x7f5d600b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:05:59.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d666a9640 1 -- 172.21.15.62:0/3672950817 >> 172.21.15.62:0/3672950817 conn(0x7f5d6009c250 msgr2=0x7f5d60072e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:05:59.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d666a9640 1 -- 172.21.15.62:0/3672950817 shutdown_connections 2024-04-18T10:05:59.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:05:59.882+0000 7f5d666a9640 1 -- 172.21.15.62:0/3672950817 wait complete. 2024-04-18T10:05:59.897 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-18T10:06:00.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.022+0000 7fd71a5f9640 1 Processor -- start 2024-04-18T10:06:00.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.022+0000 7fd71a5f9640 1 -- start start 2024-04-18T10:06:00.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd71a5f9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b4c90 0x7fd714145f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:00.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd71a5f9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71406d4f0 con 0x7fd71409e7a0 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd71a5f9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71406d7d0 con 0x7fd71406e490 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd71a5f9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd71406d2a0 con 0x7fd7140b4c90 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd718b96640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd71409e7a0 0x7fd714150410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:58658/0 (socket says 172.21.15.62:58658) 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd718b96640 1 -- 172.21.15.62:0/2427937206 learned_addr learned my addr 172.21.15.62:0/2427937206 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd713fff640 1 --2- 172.21.15.62:0/2427937206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b4c90 0x7fd714145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797808397 0 0) 0x7fd71406d4f0 con 0x7fd71409e7a0 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd6fc003530 con 0x7fd71409e7a0 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 608733692 0 0) 0x7fd6fc003530 con 0x7fd71409e7a0 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b4c90 msgr2=0x7fd714145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 --2- 172.21.15.62:0/2427937206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b4c90 0x7fd714145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 >> v1:172.21.15.145:6789/0 conn(0x7fd71406e490 legacy=0x7fd71409e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71409c570 con 0x7fd71409e7a0 2024-04-18T10:06:00.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fd70c001680 con 0x7fd71409e7a0 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 >> v1:172.21.15.62:6789/0 conn(0x7fd71409e7a0 legacy=0x7fd714150410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 --2- 172.21.15.62:0/2427937206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6fc0039f0 0x7fd6fc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 --2- 172.21.15.62:0/2427937206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b4c90 0x7fd6fc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 --2- 172.21.15.62:0/2427937206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6fc004d50 0x7fd6fc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd71406d2a0 con 0x7fd6fc0039f0 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fd6fc00bbb0 con 0x7fd6fc004d50 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd6fc00bdb0 con 0x7fd7140b4c90 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd713fff640 1 --2- 172.21.15.62:0/2427937206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6fc0039f0 0x7fd6fc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd7137fe640 1 --2- 172.21.15.62:0/2427937206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b4c90 0x7fd6fc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd718b96640 1 --2- 172.21.15.62:0/2427937206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6fc004d50 0x7fd6fc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd713fff640 1 -- 172.21.15.62:0/2427937206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b4c90 msgr2=0x7fd6fc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd713fff640 1 --2- 172.21.15.62:0/2427937206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b4c90 0x7fd6fc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd713fff640 1 -- 172.21.15.62:0/2427937206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6fc004d50 msgr2=0x7fd6fc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:00.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd713fff640 1 --2- 172.21.15.62:0/2427937206 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6fc004d50 0x7fd6fc0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:00.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.026+0000 7fd713fff640 1 -- 172.21.15.62:0/2427937206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6fc003530 con 0x7fd6fc0039f0 2024-04-18T10:06:00.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd713fff640 1 --2- 172.21.15.62:0/2427937206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6fc0039f0 0x7fd6fc003e10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd704007360 tx=0x7fd70401ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb0b6ee76b451f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:00.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7040b2070 con 0x7fd6fc0039f0 2024-04-18T10:06:00.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7040a5b20 con 0x7fd6fc0039f0 2024-04-18T10:06:00.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd712ffd640 1 -- 172.21.15.62:0/2427937206 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd704009be0 con 0x7fd6fc0039f0 2024-04-18T10:06:00.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2427937206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6fc0039f0 msgr2=0x7fd6fc003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:00.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 --2- 172.21.15.62:0/2427937206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6fc0039f0 0x7fd6fc003e10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd704007360 tx=0x7fd70401ce40 comp rx=0 tx=0).stop 2024-04-18T10:06:00.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd718b96640 1 -- 172.21.15.62:0/2427937206 reap_dead start 2024-04-18T10:06:00.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2427937206 shutdown_connections 2024-04-18T10:06:00.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2427937206 >> 172.21.15.62:0/2427937206 conn(0x7fd7140b9690 msgr2=0x7fd7140b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:00.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2427937206 shutdown_connections 2024-04-18T10:06:00.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2427937206 wait complete. 2024-04-18T10:06:00.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 Processor -- start 2024-04-18T10:06:00.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 -- start start 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6fc0039f0 0x7fd7140b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7141404a0 0x7fd7140b3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b0800 0x7fd7140b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd714151ec0 con 0x7fd7141404a0 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fd714151d40 con 0x7fd6fc0039f0 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd714152040 con 0x7fd7140b0800 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd7137fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7141404a0 0x7fd7140b3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd7137fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7141404a0 0x7fd7140b3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40814/0 (socket says 172.21.15.62:40814) 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd7137fe640 1 -- 172.21.15.62:0/2690551217 learned_addr learned my addr 172.21.15.62:0/2690551217 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd713fff640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6fc0039f0 0x7fd7140b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd718b96640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b0800 0x7fd7140b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd7137fe640 1 -- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b0800 msgr2=0x7fd7140b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:00.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd7137fe640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b0800 0x7fd7140b0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:00.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd7137fe640 1 -- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6fc0039f0 msgr2=0x7fd7140b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:00.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd7137fe640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6fc0039f0 0x7fd7140b2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:00.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd7137fe640 1 -- 172.21.15.62:0/2690551217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7141417f0 con 0x7fd7141404a0 2024-04-18T10:06:00.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd7137fe640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7141404a0 0x7fd7140b3240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd70800e500 tx=0x7fd70800e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:00.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd710ff9640 1 -- 172.21.15.62:0/2690551217 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7080053a0 con 0x7fd7141404a0 2024-04-18T10:06:00.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd710ff9640 1 -- 172.21.15.62:0/2690551217 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd708018070 con 0x7fd7141404a0 2024-04-18T10:06:00.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2690551217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71413b5a0 con 0x7fd7141404a0 2024-04-18T10:06:00.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd710ff9640 1 -- 172.21.15.62:0/2690551217 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd708003820 con 0x7fd7141404a0 2024-04-18T10:06:00.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.030+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2690551217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd714141510 con 0x7fd7141404a0 2024-04-18T10:06:00.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.034+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2690551217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd714068c00 con 0x7fd7141404a0 2024-04-18T10:06:00.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.034+0000 7fd710ff9640 1 -- 172.21.15.62:0/2690551217 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd7080039c0 con 0x7fd7141404a0 2024-04-18T10:06:00.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.034+0000 7fd710ff9640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd700068380 0x7fd70006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:00.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.034+0000 7fd710ff9640 1 -- 172.21.15.62:0/2690551217 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd7080866f0 con 0x7fd7141404a0 2024-04-18T10:06:00.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.034+0000 7fd713fff640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd700068380 0x7fd70006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:00.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.034+0000 7fd713fff640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd700068380 0x7fd70006a840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd7040af000 tx=0x7fd704004a50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:00.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.038+0000 7fd710ff9640 1 -- 172.21.15.62:0/2690551217 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd708056cc0 con 0x7fd7141404a0 2024-04-18T10:06:00.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:00.242+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2690551217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fd7140b44c0 con 0x7fd7141404a0 2024-04-18T10:06:01.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.234+0000 7fd710ff9640 1 -- 172.21.15.62:0/2690551217 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7fd7080565c0 con 0x7fd7141404a0 2024-04-18T10:06:01.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.238+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd700068380 msgr2=0x7fd70006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:01.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.238+0000 7fd71a5f9640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd700068380 0x7fd70006a840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd7040af000 tx=0x7fd704004a50 comp rx=0 tx=0).stop 2024-04-18T10:06:01.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.238+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2690551217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7141404a0 msgr2=0x7fd7140b3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:01.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.238+0000 7fd71a5f9640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7141404a0 0x7fd7140b3240 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd70800e500 tx=0x7fd70800e9c0 comp rx=0 tx=0).stop 2024-04-18T10:06:01.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.238+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2690551217 shutdown_connections 2024-04-18T10:06:01.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.238+0000 7fd71a5f9640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd700068380 0x7fd70006a840 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:01.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.238+0000 7fd71a5f9640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd7140b0800 0x7fd7140b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:01.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.238+0000 7fd71a5f9640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd7141404a0 0x7fd7140b3240 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:01.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.238+0000 7fd71a5f9640 1 --2- 172.21.15.62:0/2690551217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6fc0039f0 0x7fd7140b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:01.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.238+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2690551217 >> 172.21.15.62:0/2690551217 conn(0x7fd7140b9690 msgr2=0x7fd71409b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:01.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.238+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2690551217 shutdown_connections 2024-04-18T10:06:01.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.242+0000 7fd71a5f9640 1 -- 172.21.15.62:0/2690551217 wait complete. 2024-04-18T10:06:01.258 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-18T10:06:01.258 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-18T10:06:01.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba7144d640 1 Processor -- start 2024-04-18T10:06:01.388 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba7144d640 1 -- start start 2024-04-18T10:06:01.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba7144d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fba6c09bce0 0x7fba6c09b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:01.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba7144d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fba6c14b360 0x7fba6c14b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:01.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba7144d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba6c14bce0 con 0x7fba6c141b30 2024-04-18T10:06:01.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba7144d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fba6c0b6b10 con 0x7fba6c14b360 2024-04-18T10:06:01.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba7144d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fba6c09c0c0 con 0x7fba6c09bce0 2024-04-18T10:06:01.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba6a7fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fba6c14b360 0x7fba6c14b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:01.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba6a7fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fba6c14b360 0x7fba6c14b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53020/0 (socket says 172.21.15.62:53020) 2024-04-18T10:06:01.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba6a7fc640 1 -- 172.21.15.62:0/4118691407 learned_addr learned my addr 172.21.15.62:0/4118691407 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:01.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba6a7fc640 1 -- 172.21.15.62:0/4118691407 >> v2:172.21.15.145:3300/0 conn(0x7fba6c09bce0 msgr2=0x7fba6c09b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:01.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba6affd640 1 --2- 172.21.15.62:0/4118691407 >> v2:172.21.15.145:3300/0 conn(0x7fba6c09bce0 0x7fba6c09b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:01.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba6a7fc640 1 --2- 172.21.15.62:0/4118691407 >> v2:172.21.15.145:3300/0 conn(0x7fba6c09bce0 0x7fba6c09b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:01.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba6a7fc640 1 -- 172.21.15.62:0/4118691407 >> v1:172.21.15.145:6789/0 conn(0x7fba6c141b30 legacy=0x7fba6c141fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:01.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba6a7fc640 1 -- 172.21.15.62:0/4118691407 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6c0b64c0 con 0x7fba6c14b360 2024-04-18T10:06:01.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.390+0000 7fba6affd640 1 --2- 172.21.15.62:0/4118691407 >> v2:172.21.15.145:3300/0 conn(0x7fba6c09bce0 0x7fba6c09b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:01.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba6a7fc640 1 --2- 172.21.15.62:0/4118691407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba6c14b360 0x7fba6c14b7a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0b80e0 tx=0x7fba5401c1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d217e7497646dfbd server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:01.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba69ffb640 1 -- 172.21.15.62:0/4118691407 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba540b0070 con 0x7fba6c14b360 2024-04-18T10:06:01.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba69ffb640 1 -- 172.21.15.62:0/4118691407 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba540a4e50 con 0x7fba6c14b360 2024-04-18T10:06:01.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba69ffb640 1 -- 172.21.15.62:0/4118691407 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba540ab820 con 0x7fba6c14b360 2024-04-18T10:06:01.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 -- 172.21.15.62:0/4118691407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba6c14b360 msgr2=0x7fba6c14b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:01.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 --2- 172.21.15.62:0/4118691407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba6c14b360 0x7fba6c14b7a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0b80e0 tx=0x7fba5401c1e0 comp rx=0 tx=0).stop 2024-04-18T10:06:01.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 -- 172.21.15.62:0/4118691407 shutdown_connections 2024-04-18T10:06:01.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 --2- 172.21.15.62:0/4118691407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba6c14b360 0x7fba6c14b7a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:01.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 --2- 172.21.15.62:0/4118691407 >> v2:172.21.15.145:3300/0 conn(0x7fba6c09bce0 0x7fba6c09b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:01.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 -- 172.21.15.62:0/4118691407 >> 172.21.15.62:0/4118691407 conn(0x7fba6c09c360 msgr2=0x7fba6c09c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:01.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 -- 172.21.15.62:0/4118691407 shutdown_connections 2024-04-18T10:06:01.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 -- 172.21.15.62:0/4118691407 wait complete. 2024-04-18T10:06:01.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 Processor -- start 2024-04-18T10:06:01.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 -- start start 2024-04-18T10:06:01.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba6c09bce0 0x7fba6c145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:01.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba6c141b30 0x7fba6c143c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:01.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba6c14b360 0x7fba6c1e34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:01.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c154eb0 con 0x7fba6c14b360 2024-04-18T10:06:01.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fba6c09b810 con 0x7fba6c09bce0 2024-04-18T10:06:01.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba7144d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fba6c155030 con 0x7fba6c141b30 2024-04-18T10:06:01.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba6b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba6c14b360 0x7fba6c1e34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:01.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba6b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba6c14b360 0x7fba6c1e34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53026/0 (socket says 172.21.15.62:53026) 2024-04-18T10:06:01.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.394+0000 7fba6b7fe640 1 -- 172.21.15.62:0/2784193637 learned_addr learned my addr 172.21.15.62:0/2784193637 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:01.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba6affd640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba6c09bce0 0x7fba6c145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:01.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba6a7fc640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba6c141b30 0x7fba6c143c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:01.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba6b7fe640 1 -- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba6c141b30 msgr2=0x7fba6c143c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:01.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba6b7fe640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba6c141b30 0x7fba6c143c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:01.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba6b7fe640 1 -- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba6c09bce0 msgr2=0x7fba6c145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:01.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba6b7fe640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba6c09bce0 0x7fba6c145160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:01.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba6b7fe640 1 -- 172.21.15.62:0/2784193637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba6c09ef40 con 0x7fba6c14b360 2024-04-18T10:06:01.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba6a7fc640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba6c141b30 0x7fba6c143c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:01.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba6b7fe640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba6c14b360 0x7fba6c1e34f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fba6400c7b0 tx=0x7fba6400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:01.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba637fe640 1 -- 172.21.15.62:0/2784193637 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6400ce60 con 0x7fba6c14b360 2024-04-18T10:06:01.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba637fe640 1 -- 172.21.15.62:0/2784193637 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba64004b20 con 0x7fba6c14b360 2024-04-18T10:06:01.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba7144d640 1 -- 172.21.15.62:0/2784193637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba6c144700 con 0x7fba6c14b360 2024-04-18T10:06:01.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba637fe640 1 -- 172.21.15.62:0/2784193637 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba6400a850 con 0x7fba6c14b360 2024-04-18T10:06:01.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba7144d640 1 -- 172.21.15.62:0/2784193637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba6c144150 con 0x7fba6c14b360 2024-04-18T10:06:01.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba7144d640 1 -- 172.21.15.62:0/2784193637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba6c145870 con 0x7fba6c14b360 2024-04-18T10:06:01.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba637fe640 1 -- 172.21.15.62:0/2784193637 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fba6400a9f0 con 0x7fba6c14b360 2024-04-18T10:06:01.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba637fe640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fba4c068380 0x7fba4c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:01.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba637fe640 1 -- 172.21.15.62:0/2784193637 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fba64086ac0 con 0x7fba6c14b360 2024-04-18T10:06:01.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.398+0000 7fba6affd640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fba4c068380 0x7fba4c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:01.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.402+0000 7fba6affd640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fba4c068380 0x7fba4c06a840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fba5c00a9a0 tx=0x7fba5c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:01.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.402+0000 7fba637fe640 1 -- 172.21.15.62:0/2784193637 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fba64020070 con 0x7fba6c14b360 2024-04-18T10:06:01.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:01.602+0000 7fba7144d640 1 -- 172.21.15.62:0/2784193637 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fba6c068c00 con 0x7fba6c14b360 2024-04-18T10:06:02.237 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.238+0000 7fba637fe640 1 -- 172.21.15.62:0/2784193637 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7fba64057110 con 0x7fba6c14b360 2024-04-18T10:06:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.242+0000 7fba7144d640 1 -- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fba4c068380 msgr2=0x7fba4c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.242+0000 7fba7144d640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fba4c068380 0x7fba4c06a840 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fba5c00a9a0 tx=0x7fba5c008040 comp rx=0 tx=0).stop 2024-04-18T10:06:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.242+0000 7fba7144d640 1 -- 172.21.15.62:0/2784193637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba6c14b360 msgr2=0x7fba6c1e34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.242+0000 7fba7144d640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba6c14b360 0x7fba6c1e34f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fba6400c7b0 tx=0x7fba6400cc70 comp rx=0 tx=0).stop 2024-04-18T10:06:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.242+0000 7fba7144d640 1 -- 172.21.15.62:0/2784193637 shutdown_connections 2024-04-18T10:06:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.242+0000 7fba7144d640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fba4c068380 0x7fba4c06a840 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.242+0000 7fba7144d640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba6c14b360 0x7fba6c1e34f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.242+0000 7fba7144d640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba6c141b30 0x7fba6c143c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.242+0000 7fba7144d640 1 --2- 172.21.15.62:0/2784193637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba6c09bce0 0x7fba6c145160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.242+0000 7fba7144d640 1 -- 172.21.15.62:0/2784193637 >> 172.21.15.62:0/2784193637 conn(0x7fba6c09c360 msgr2=0x7fba6c0b8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.242+0000 7fba7144d640 1 -- 172.21.15.62:0/2784193637 shutdown_connections 2024-04-18T10:06:02.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.242+0000 7fba7144d640 1 -- 172.21.15.62:0/2784193637 wait complete. 2024-04-18T10:06:02.259 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-18T10:06:02.260 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:02.281 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-04-18T10:06:02.278+0000 7f66aea66640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-18T10:06:02.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cd0fff640 1 Processor -- start 2024-04-18T10:06:02.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cd0fff640 1 -- start start 2024-04-18T10:06:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cd0fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0b9d90 0x7f7ccc1450e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cd0fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7ccc0b44e0 0x7f7ccc0b4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cd0fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7ccc148130 0x7f7ccc1485c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cd0fff640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f7ccc145b80 con 0x7f7ccc0b44e0 2024-04-18T10:06:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cd0fff640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f7ccc145620 con 0x7f7ccc148130 2024-04-18T10:06:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cd0fff640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7ccc0ba170 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cc9d9b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7ccc0b44e0 0x7f7ccc0b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cc9d9b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7ccc0b44e0 0x7f7ccc0b4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53036/0 (socket says 172.21.15.62:53036) 2024-04-18T10:06:02.389 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cc9d9b640 1 -- 172.21.15.62:0/3902128646 learned_addr learned my addr 172.21.15.62:0/3902128646 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:02.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7ccad9d640 1 --2- 172.21.15.62:0/3902128646 >> v2:172.21.15.145:3300/0 conn(0x7f7ccc148130 0x7f7ccc1485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:02.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cca59c640 1 --2- 172.21.15.62:0/3902128646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0b9d90 0x7f7ccc1450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:02.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cc9d9b640 1 -- 172.21.15.62:0/3902128646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0b9d90 msgr2=0x7f7ccc1450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:02.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cc9d9b640 1 --2- 172.21.15.62:0/3902128646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0b9d90 0x7f7ccc1450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cc9d9b640 1 -- 172.21.15.62:0/3902128646 >> v2:172.21.15.145:3300/0 conn(0x7f7ccc148130 msgr2=0x7f7ccc1485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:02.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cc9d9b640 1 --2- 172.21.15.62:0/3902128646 >> v2:172.21.15.145:3300/0 conn(0x7f7ccc148130 0x7f7ccc1485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cc9d9b640 1 -- 172.21.15.62:0/3902128646 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc1458a0 con 0x7f7ccc0b44e0 2024-04-18T10:06:02.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7cca59c640 1 --2- 172.21.15.62:0/3902128646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0b9d90 0x7f7ccc1450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:02.390 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.390+0000 7f7ccad9d640 1 --2- 172.21.15.62:0/3902128646 >> v2:172.21.15.145:3300/0 conn(0x7f7ccc148130 0x7f7ccc1485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:02.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc9d9b640 1 --2- 172.21.15.62:0/3902128646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ccc0b44e0 0x7f7ccc0b4940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0ae980 tx=0x7f7cc001e6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2330a0560e1b0241 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:02.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc959a640 1 -- 172.21.15.62:0/3902128646 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc00a9070 con 0x7f7ccc0b44e0 2024-04-18T10:06:02.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc959a640 1 -- 172.21.15.62:0/3902128646 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cc00b2070 con 0x7f7ccc0b44e0 2024-04-18T10:06:02.391 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc959a640 1 -- 172.21.15.62:0/3902128646 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc00ad890 con 0x7f7ccc0b44e0 2024-04-18T10:06:02.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 -- 172.21.15.62:0/3902128646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ccc0b44e0 msgr2=0x7f7ccc0b4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 --2- 172.21.15.62:0/3902128646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ccc0b44e0 0x7f7ccc0b4940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc0ae980 tx=0x7f7cc001e6a0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 -- 172.21.15.62:0/3902128646 shutdown_connections 2024-04-18T10:06:02.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 --2- 172.21.15.62:0/3902128646 >> v2:172.21.15.145:3300/0 conn(0x7f7ccc148130 0x7f7ccc1485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 --2- 172.21.15.62:0/3902128646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ccc0b44e0 0x7f7ccc0b4940 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.392 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 --2- 172.21.15.62:0/3902128646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc0b9d90 0x7f7ccc1450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 -- 172.21.15.62:0/3902128646 >> 172.21.15.62:0/3902128646 conn(0x7f7ccc06c0d0 msgr2=0x7f7ccc0b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:02.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 -- 172.21.15.62:0/3902128646 shutdown_connections 2024-04-18T10:06:02.393 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 -- 172.21.15.62:0/3902128646 wait complete. 2024-04-18T10:06:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 Processor -- start 2024-04-18T10:06:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 -- start start 2024-04-18T10:06:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ccc0b44e0 0x7f7ccc072150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ccc0b9d90 0x7f7ccc14c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc148130 0x7f7ccc1df7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc09e3d0 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7ccc1443c0 con 0x7f7ccc0b44e0 2024-04-18T10:06:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7ccc14f2f0 con 0x7f7ccc148130 2024-04-18T10:06:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ccc0b9d90 0x7f7ccc14c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ccc0b9d90 0x7f7ccc14c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53038/0 (socket says 172.21.15.62:53038) 2024-04-18T10:06:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc9d9b640 1 -- 172.21.15.62:0/1685511538 learned_addr learned my addr 172.21.15.62:0/1685511538 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:02.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cca59c640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ccc0b44e0 0x7f7ccc072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc9d9b640 1 -- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc148130 msgr2=0x7f7ccc1df7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc9d9b640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc148130 0x7f7ccc1df7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc9d9b640 1 -- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ccc0b44e0 msgr2=0x7f7ccc072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc9d9b640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ccc0b44e0 0x7f7ccc072150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc9d9b640 1 -- 172.21.15.62:0/1685511538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ccc14ab90 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cca59c640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ccc0b44e0 0x7f7ccc072150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cc9d9b640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ccc0b9d90 0x7f7ccc14c510 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7cc001cd50 tx=0x7f7cc001b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cbaffd640 1 -- 172.21.15.62:0/1685511538 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc00b6800 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cbaffd640 1 -- 172.21.15.62:0/1685511538 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cc00b69a0 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 -- 172.21.15.62:0/1685511538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ccc14ca50 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cbaffd640 1 -- 172.21.15.62:0/1685511538 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc00b6d90 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.394+0000 7f7cd0fff640 1 -- 172.21.15.62:0/1685511538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ccc14a820 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.398+0000 7f7cd0fff640 1 -- 172.21.15.62:0/1685511538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ccc14f470 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.398+0000 7f7cbaffd640 1 -- 172.21.15.62:0/1685511538 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7cc00c9440 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.398+0000 7f7cbaffd640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7cac068380 0x7f7cac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.398+0000 7f7cbaffd640 1 -- 172.21.15.62:0/1685511538 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7cc01234f0 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.398+0000 7f7cca59c640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7cac068380 0x7f7cac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:02.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.398+0000 7f7cca59c640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7cac068380 0x7f7cac06a840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc008930 tx=0x7f7cbc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:02.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.402+0000 7f7cbaffd640 1 -- 172.21.15.62:0/1685511538 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7cc0018320 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.594+0000 7f7cd0fff640 1 -- 172.21.15.62:0/1685511538 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ccc068b90 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.594 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.594+0000 7f7cbaffd640 1 -- 172.21.15.62:0/1685511538 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3527 (secure 0 0 0) 0x7f7cc00f3b40 con 0x7f7ccc0b9d90 2024-04-18T10:06:02.594 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:02.594 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/904929673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":904929673},{"type":"v1","addr":"172.21.15.145:6835","nonce":904929673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:05:58.588100+0000","modified":"2024-04-18T10:06:01.605447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2120622858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2120622858},{"type":"v1","addr":"172.21.15.62:6833","nonce":2120622858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-18T10:06:02.595 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-04-18T10:06:02.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.598+0000 7f7cd0fff640 1 -- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7cac068380 msgr2=0x7f7cac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.598+0000 7f7cd0fff640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7cac068380 0x7f7cac06a840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc008930 tx=0x7f7cbc008040 comp rx=0 tx=0).stop 2024-04-18T10:06:02.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.598+0000 7f7cd0fff640 1 -- 172.21.15.62:0/1685511538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ccc0b9d90 msgr2=0x7f7ccc14c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.598+0000 7f7cd0fff640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ccc0b9d90 0x7f7ccc14c510 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7cc001cd50 tx=0x7f7cc001b440 comp rx=0 tx=0).stop 2024-04-18T10:06:02.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.598+0000 7f7cd0fff640 1 -- 172.21.15.62:0/1685511538 shutdown_connections 2024-04-18T10:06:02.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.598+0000 7f7cd0fff640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7cac068380 0x7f7cac06a840 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.598+0000 7f7cd0fff640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ccc148130 0x7f7ccc1df7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.598+0000 7f7cd0fff640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ccc0b9d90 0x7f7ccc14c510 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.598+0000 7f7cd0fff640 1 --2- 172.21.15.62:0/1685511538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ccc0b44e0 0x7f7ccc072150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.598+0000 7f7cd0fff640 1 -- 172.21.15.62:0/1685511538 >> 172.21.15.62:0/1685511538 conn(0x7f7ccc06c0d0 msgr2=0x7f7ccc0af930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:02.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.598+0000 7f7cd0fff640 1 -- 172.21.15.62:0/1685511538 shutdown_connections 2024-04-18T10:06:02.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.598+0000 7f7cd0fff640 1 -- 172.21.15.62:0/1685511538 wait complete. 2024-04-18T10:06:02.611 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:02.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.778+0000 7f2bbf9b6640 1 Processor -- start 2024-04-18T10:06:02.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.778+0000 7f2bbf9b6640 1 -- start start 2024-04-18T10:06:02.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbf9b6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2bb80b7f70 0x7f2bb8141900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbf9b6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2bb8141e40 0x7f2bb81525b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbf9b6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bb80b6740 con 0x7f2bb80b6360 2024-04-18T10:06:02.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbf9b6640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f2bb80b7720 con 0x7f2bb80b7f70 2024-04-18T10:06:02.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbf9b6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f2bb8147e40 con 0x7f2bb8141e40 2024-04-18T10:06:02.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbcf51640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2bb80b7f70 0x7f2bb8141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:02.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbd752640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2bb80b6360 0x7f2bb8148640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54876/0 (socket says 172.21.15.62:54876) 2024-04-18T10:06:02.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbd752640 1 -- 172.21.15.62:0/557499933 learned_addr learned my addr 172.21.15.62:0/557499933 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:02.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbdf53640 1 --2- 172.21.15.62:0/557499933 >> v2:172.21.15.145:3300/0 conn(0x7f2bb8141e40 0x7f2bb81525b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:02.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbcf51640 1 -- 172.21.15.62:0/557499933 >> v2:172.21.15.145:3300/0 conn(0x7f2bb8141e40 msgr2=0x7f2bb81525b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:02.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbcf51640 1 --2- 172.21.15.62:0/557499933 >> v2:172.21.15.145:3300/0 conn(0x7f2bb8141e40 0x7f2bb81525b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbcf51640 1 -- 172.21.15.62:0/557499933 >> v1:172.21.15.62:6789/0 conn(0x7f2bb80b6360 legacy=0x7f2bb8148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbcf51640 1 -- 172.21.15.62:0/557499933 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb8148120 con 0x7f2bb80b7f70 2024-04-18T10:06:02.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2bbcf51640 1 --2- 172.21.15.62:0/557499933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bb80b7f70 0x7f2bb8141900 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2bb808c130 tx=0x7f2ba801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f82eebd1c1bf4f server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:02.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2baffff640 1 -- 172.21.15.62:0/557499933 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80b0070 con 0x7f2bb80b7f70 2024-04-18T10:06:02.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2baffff640 1 -- 172.21.15.62:0/557499933 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ba80ab600 con 0x7f2bb80b7f70 2024-04-18T10:06:02.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.782+0000 7f2baffff640 1 -- 172.21.15.62:0/557499933 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ba80ab9d0 con 0x7f2bb80b7f70 2024-04-18T10:06:02.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/557499933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bb80b7f70 msgr2=0x7f2bb8141900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 --2- 172.21.15.62:0/557499933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bb80b7f70 0x7f2bb8141900 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2bb808c130 tx=0x7f2ba801c3f0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/557499933 shutdown_connections 2024-04-18T10:06:02.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 --2- 172.21.15.62:0/557499933 >> v2:172.21.15.145:3300/0 conn(0x7f2bb8141e40 0x7f2bb81525b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 --2- 172.21.15.62:0/557499933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bb80b7f70 0x7f2bb8141900 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/557499933 >> 172.21.15.62:0/557499933 conn(0x7f2bb80ba070 msgr2=0x7f2bb80ba480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:02.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/557499933 shutdown_connections 2024-04-18T10:06:02.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/557499933 wait complete. 2024-04-18T10:06:02.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 Processor -- start 2024-04-18T10:06:02.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 -- start start 2024-04-18T10:06:02.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bb80b6360 0x7f2bb8134920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2bb80b7f70 0x7f2bb8146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bb8141e40 0x7f2bb813a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb80b7240 con 0x7f2bb8141e40 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f2bb80b73c0 con 0x7f2bb80b7f70 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2bb8155310 con 0x7f2bb80b6360 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbdf53640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bb8141e40 0x7f2bb813a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbdf53640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bb8141e40 0x7f2bb813a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53064/0 (socket says 172.21.15.62:53064) 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbdf53640 1 -- 172.21.15.62:0/4218822909 learned_addr learned my addr 172.21.15.62:0/4218822909 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbcf51640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2bb80b7f70 0x7f2bb8146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbd752640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bb80b6360 0x7f2bb8134920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbdf53640 1 -- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bb80b6360 msgr2=0x7f2bb8134920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbdf53640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bb80b6360 0x7f2bb8134920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbdf53640 1 -- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2bb80b7f70 msgr2=0x7f2bb8146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbdf53640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2bb80b7f70 0x7f2bb8146a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbdf53640 1 -- 172.21.15.62:0/4218822909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bb8149040 con 0x7f2bb8141e40 2024-04-18T10:06:02.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbd752640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bb80b6360 0x7f2bb8134920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:02.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbcf51640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2bb80b7f70 0x7f2bb8146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:02.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbdf53640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bb8141e40 0x7f2bb813a380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0007e90 tx=0x7f2bb000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:02.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2badffb640 1 -- 172.21.15.62:0/4218822909 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0014420 con 0x7f2bb8141e40 2024-04-18T10:06:02.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2badffb640 1 -- 172.21.15.62:0/4218822909 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bb0019070 con 0x7f2bb8141e40 2024-04-18T10:06:02.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/4218822909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bb8149320 con 0x7f2bb8141e40 2024-04-18T10:06:02.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2badffb640 1 -- 172.21.15.62:0/4218822909 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb0014800 con 0x7f2bb8141e40 2024-04-18T10:06:02.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/4218822909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bb814a850 con 0x7f2bb8141e40 2024-04-18T10:06:02.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.786+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/4218822909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bb8069230 con 0x7f2bb8141e40 2024-04-18T10:06:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.790+0000 7f2badffb640 1 -- 172.21.15.62:0/4218822909 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2bb00149a0 con 0x7f2bb8141e40 2024-04-18T10:06:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.790+0000 7f2badffb640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2b9c068380 0x7f2b9c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.790+0000 7f2badffb640 1 -- 172.21.15.62:0/4218822909 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2bb0087190 con 0x7f2bb8141e40 2024-04-18T10:06:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.790+0000 7f2bbd752640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2b9c068380 0x7f2b9c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.790+0000 7f2bbd752640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2b9c068380 0x7f2b9c06a840 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4007fd0 tx=0x7f2bb4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:02.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.794+0000 7f2badffb640 1 -- 172.21.15.62:0/4218822909 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2bb0011260 con 0x7f2bb8141e40 2024-04-18T10:06:02.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.958+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/4218822909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bb813a030 con 0x7f2bb8141e40 2024-04-18T10:06:02.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.958+0000 7f2badffb640 1 -- 172.21.15.62:0/4218822909 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f2bb00577e0 con 0x7f2bb8141e40 2024-04-18T10:06:02.959 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:02.959 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:02.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.962+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2b9c068380 msgr2=0x7f2b9c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.962+0000 7f2bbf9b6640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2b9c068380 0x7f2b9c06a840 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4007fd0 tx=0x7f2bb4009040 comp rx=0 tx=0).stop 2024-04-18T10:06:02.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.962+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/4218822909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bb8141e40 msgr2=0x7f2bb813a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:02.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.962+0000 7f2bbf9b6640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bb8141e40 0x7f2bb813a380 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2bb0007e90 tx=0x7f2bb000e4b0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.962+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/4218822909 shutdown_connections 2024-04-18T10:06:02.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.962+0000 7f2bbf9b6640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2b9c068380 0x7f2b9c06a840 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.962+0000 7f2bbf9b6640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2bb8141e40 0x7f2bb813a380 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.962+0000 7f2bbf9b6640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2bb80b7f70 0x7f2bb8146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.962+0000 7f2bbf9b6640 1 --2- 172.21.15.62:0/4218822909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2bb80b6360 0x7f2bb8134920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:02.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.962+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/4218822909 >> 172.21.15.62:0/4218822909 conn(0x7f2bb80ba070 msgr2=0x7f2bb813b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:02.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.962+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/4218822909 shutdown_connections 2024-04-18T10:06:02.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:02.962+0000 7f2bbf9b6640 1 -- 172.21.15.62:0/4218822909 wait complete. 2024-04-18T10:06:02.978 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-18T10:06:02.978 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T10:06:03.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.102+0000 7f2ce19c0640 1 Processor -- start 2024-04-18T10:06:03.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.102+0000 7f2ce19c0640 1 -- start start 2024-04-18T10:06:03.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2ce19c0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2cdc0b58d0 0x7f2cdc0b7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2ce19c0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cdc142040 0x7f2cdc1526c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2ce19c0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cdc147d40 con 0x7f2cdc148aa0 2024-04-18T10:06:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2ce19c0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f2cdc0b7800 con 0x7f2cdc0b58d0 2024-04-18T10:06:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2ce19c0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc0b8510 con 0x7f2cdc142040 2024-04-18T10:06:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cdaffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2cdc0b58d0 0x7f2cdc0b7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cda7fc640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2cdc148aa0 0x7f2cdc141930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54880/0 (socket says 172.21.15.62:54880) 2024-04-18T10:06:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cda7fc640 1 -- 172.21.15.62:0/524526430 learned_addr learned my addr 172.21.15.62:0/524526430 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cdb7fe640 1 --2- 172.21.15.62:0/524526430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cdc142040 0x7f2cdc1526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cdaffd640 1 -- 172.21.15.62:0/524526430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cdc142040 msgr2=0x7f2cdc1526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cd9ffb640 1 -- 172.21.15.62:0/524526430 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3936852419 0 0) 0x7f2cd0001120 con 0x7f2cdc148aa0 2024-04-18T10:06:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cdaffd640 1 --2- 172.21.15.62:0/524526430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cdc142040 0x7f2cdc1526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cdaffd640 1 -- 172.21.15.62:0/524526430 >> v1:172.21.15.62:6789/0 conn(0x7f2cdc148aa0 legacy=0x7f2cdc141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cdaffd640 1 -- 172.21.15.62:0/524526430 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc0b5cb0 con 0x7f2cdc0b58d0 2024-04-18T10:06:03.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cdb7fe640 1 --2- 172.21.15.62:0/524526430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cdc142040 0x7f2cdc1526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:03.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cdaffd640 1 --2- 172.21.15.62:0/524526430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cdc0b58d0 0x7f2cdc0b7fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0097c0 tx=0x7f2ccc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7619c7eb54c30df6 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:03.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cd9ffb640 1 -- 172.21.15.62:0/524526430 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc0b1070 con 0x7f2cdc0b58d0 2024-04-18T10:06:03.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cd9ffb640 1 -- 172.21.15.62:0/524526430 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ccc0a8050 con 0x7f2cdc0b58d0 2024-04-18T10:06:03.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.106+0000 7f2cd9ffb640 1 -- 172.21.15.62:0/524526430 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc0ac860 con 0x7f2cdc0b58d0 2024-04-18T10:06:03.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 -- 172.21.15.62:0/524526430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cdc0b58d0 msgr2=0x7f2cdc0b7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 --2- 172.21.15.62:0/524526430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cdc0b58d0 0x7f2cdc0b7fd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0097c0 tx=0x7f2ccc01c3e0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 -- 172.21.15.62:0/524526430 shutdown_connections 2024-04-18T10:06:03.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 --2- 172.21.15.62:0/524526430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cdc142040 0x7f2cdc1526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 --2- 172.21.15.62:0/524526430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cdc0b58d0 0x7f2cdc0b7fd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 -- 172.21.15.62:0/524526430 >> 172.21.15.62:0/524526430 conn(0x7f2cdc0ba0d0 msgr2=0x7f2cdc0ba4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:03.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 -- 172.21.15.62:0/524526430 shutdown_connections 2024-04-18T10:06:03.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 -- 172.21.15.62:0/524526430 wait complete. 2024-04-18T10:06:03.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 Processor -- start 2024-04-18T10:06:03.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 -- start start 2024-04-18T10:06:03.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cdc0b58d0 0x7f2cdc14b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cdc142040 0x7f2cdc14a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cdc148aa0 0x7f2cdc1e0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc0b6870 con 0x7f2cdc142040 2024-04-18T10:06:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f2cdc0b7480 con 0x7f2cdc148aa0 2024-04-18T10:06:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2cdc0b69f0 con 0x7f2cdc0b58d0 2024-04-18T10:06:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cda7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cdc142040 0x7f2cdc14a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cda7fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cdc142040 0x7f2cdc14a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53080/0 (socket says 172.21.15.62:53080) 2024-04-18T10:06:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cda7fc640 1 -- 172.21.15.62:0/4022547355 learned_addr learned my addr 172.21.15.62:0/4022547355 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cdaffd640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cdc0b58d0 0x7f2cdc14b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cdb7fe640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cdc148aa0 0x7f2cdc1e0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cda7fc640 1 -- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cdc0b58d0 msgr2=0x7f2cdc14b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cda7fc640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cdc0b58d0 0x7f2cdc14b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cda7fc640 1 -- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cdc148aa0 msgr2=0x7f2cdc1e0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cda7fc640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cdc148aa0 0x7f2cdc1e0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cda7fc640 1 -- 172.21.15.62:0/4022547355 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cdc0b9890 con 0x7f2cdc142040 2024-04-18T10:06:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cda7fc640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cdc142040 0x7f2cdc14a0c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000c450 tx=0x7f2cd000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cc3fff640 1 -- 172.21.15.62:0/4022547355 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0018070 con 0x7f2cdc142040 2024-04-18T10:06:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cc3fff640 1 -- 172.21.15.62:0/4022547355 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd00074e0 con 0x7f2cdc142040 2024-04-18T10:06:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 -- 172.21.15.62:0/4022547355 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cdc0b8340 con 0x7f2cdc142040 2024-04-18T10:06:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2cc3fff640 1 -- 172.21.15.62:0/4022547355 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd00049f0 con 0x7f2cdc142040 2024-04-18T10:06:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 -- 172.21.15.62:0/4022547355 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cdc142e40 con 0x7f2cdc142040 2024-04-18T10:06:03.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.110+0000 7f2ce19c0640 1 -- 172.21.15.62:0/4022547355 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cdc143230 con 0x7f2cdc142040 2024-04-18T10:06:03.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.114+0000 7f2cc3fff640 1 -- 172.21.15.62:0/4022547355 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2cd001b070 con 0x7f2cdc142040 2024-04-18T10:06:03.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.114+0000 7f2cc3fff640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2cc8068380 0x7f2cc806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.114+0000 7f2cc3fff640 1 -- 172.21.15.62:0/4022547355 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2cd0085ee0 con 0x7f2cdc142040 2024-04-18T10:06:03.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.114+0000 7f2cdaffd640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2cc8068380 0x7f2cc806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.114+0000 7f2cdaffd640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2cc8068380 0x7f2cc806a840 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0ae000 tx=0x7f2ccc0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:03.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.118+0000 7f2cc3fff640 1 -- 172.21.15.62:0/4022547355 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2cd0022080 con 0x7f2cdc142040 2024-04-18T10:06:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.278+0000 7f2ce19c0640 1 -- 172.21.15.62:0/4022547355 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2cdc068b90 con 0x7f2cc8068380 2024-04-18T10:06:03.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.298+0000 7f2cc3fff640 1 -- 172.21.15.62:0/4022547355 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251955 (secure 0 0 0) 0x7f2cdc068b90 con 0x7f2cc8068380 2024-04-18T10:06:03.300 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:03.301 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-18T10:06:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.302+0000 7f2ce19c0640 1 -- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2cc8068380 msgr2=0x7f2cc806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.302+0000 7f2ce19c0640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2cc8068380 0x7f2cc806a840 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0ae000 tx=0x7f2ccc0a7000 comp rx=0 tx=0).stop 2024-04-18T10:06:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.302+0000 7f2ce19c0640 1 -- 172.21.15.62:0/4022547355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cdc142040 msgr2=0x7f2cdc14a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.302+0000 7f2ce19c0640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cdc142040 0x7f2cdc14a0c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000c450 tx=0x7f2cd000c910 comp rx=0 tx=0).stop 2024-04-18T10:06:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.302+0000 7f2ce19c0640 1 -- 172.21.15.62:0/4022547355 shutdown_connections 2024-04-18T10:06:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.302+0000 7f2ce19c0640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2cc8068380 0x7f2cc806a840 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.302+0000 7f2ce19c0640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cdc148aa0 0x7f2cdc1e0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.302+0000 7f2ce19c0640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2cdc142040 0x7f2cdc14a0c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.302+0000 7f2ce19c0640 1 --2- 172.21.15.62:0/4022547355 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cdc0b58d0 0x7f2cdc14b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.302+0000 7f2ce19c0640 1 -- 172.21.15.62:0/4022547355 >> 172.21.15.62:0/4022547355 conn(0x7f2cdc0ba0d0 msgr2=0x7f2cdc1403e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.302+0000 7f2ce19c0640 1 -- 172.21.15.62:0/4022547355 shutdown_connections 2024-04-18T10:06:03.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.302+0000 7f2ce19c0640 1 -- 172.21.15.62:0/4022547355 wait complete. 2024-04-18T10:06:03.318 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-18T10:06:03.126165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219116,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755604,"statfs":{"total":773094113280,"available":772869738496,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.931952"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242380+0000","last_change":"2024-04-18T10:05:59.270038+0000","last_active":"2024-04-18T10:06:00.242380+0000","last_peered":"2024-04-18T10:06:00.242380+0000","last_clean":"2024-04-18T10:06:00.242380+0000","last_became_active":"2024-04-18T10:05:59.269226+0000","last_became_peered":"2024-04-18T10:05:59.269226+0000","last_unstale":"2024-04-18T10:06:00.242380+0000","last_undegraded":"2024-04-18T10:06:00.242380+0000","last_fullsized":"2024-04-18T10:06:00.242380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:45:43.670227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232794+0000","last_change":"2024-04-18T10:05:59.230274+0000","last_active":"2024-04-18T10:06:00.232794+0000","last_peered":"2024-04-18T10:06:00.232794+0000","last_clean":"2024-04-18T10:06:00.232794+0000","last_became_active":"2024-04-18T10:05:59.229843+0000","last_became_peered":"2024-04-18T10:05:59.229843+0000","last_unstale":"2024-04-18T10:06:00.232794+0000","last_undegraded":"2024-04-18T10:06:00.232794+0000","last_fullsized":"2024-04-18T10:06:00.232794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384628+0000","last_change":"2024-04-18T10:05:59.228858+0000","last_active":"2024-04-18T10:06:00.384628+0000","last_peered":"2024-04-18T10:06:00.384628+0000","last_clean":"2024-04-18T10:06:00.384628+0000","last_became_active":"2024-04-18T10:05:59.228382+0000","last_became_peered":"2024-04-18T10:05:59.228382+0000","last_unstale":"2024-04-18T10:06:00.384628+0000","last_undegraded":"2024-04-18T10:06:00.384628+0000","last_fullsized":"2024-04-18T10:06:00.384628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240458+0000","last_change":"2024-04-18T10:05:59.240142+0000","last_active":"2024-04-18T10:06:00.240458+0000","last_peered":"2024-04-18T10:06:00.240458+0000","last_clean":"2024-04-18T10:06:00.240458+0000","last_became_active":"2024-04-18T10:05:59.239298+0000","last_became_peered":"2024-04-18T10:05:59.239298+0000","last_unstale":"2024-04-18T10:06:00.240458+0000","last_undegraded":"2024-04-18T10:06:00.240458+0000","last_fullsized":"2024-04-18T10:06:00.240458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:49.354229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239594+0000","last_change":"2024-04-18T10:05:59.235434+0000","last_active":"2024-04-18T10:06:00.239594+0000","last_peered":"2024-04-18T10:06:00.239594+0000","last_clean":"2024-04-18T10:06:00.239594+0000","last_became_active":"2024-04-18T10:05:59.234575+0000","last_became_peered":"2024-04-18T10:05:59.234575+0000","last_unstale":"2024-04-18T10:06:00.239594+0000","last_undegraded":"2024-04-18T10:06:00.239594+0000","last_fullsized":"2024-04-18T10:06:00.239594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:34:28.871228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238452+0000","last_change":"2024-04-18T10:05:59.232083+0000","last_active":"2024-04-18T10:06:00.238452+0000","last_peered":"2024-04-18T10:06:00.238452+0000","last_clean":"2024-04-18T10:06:00.238452+0000","last_became_active":"2024-04-18T10:05:59.231313+0000","last_became_peered":"2024-04-18T10:05:59.231313+0000","last_unstale":"2024-04-18T10:06:00.238452+0000","last_undegraded":"2024-04-18T10:06:00.238452+0000","last_fullsized":"2024-04-18T10:06:00.238452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238876+0000","last_change":"2024-04-18T10:05:59.250962+0000","last_active":"2024-04-18T10:06:00.238876+0000","last_peered":"2024-04-18T10:06:00.238876+0000","last_clean":"2024-04-18T10:06:00.238876+0000","last_became_active":"2024-04-18T10:05:59.250059+0000","last_became_peered":"2024-04-18T10:05:59.250059+0000","last_unstale":"2024-04-18T10:06:00.238876+0000","last_undegraded":"2024-04-18T10:06:00.238876+0000","last_fullsized":"2024-04-18T10:06:00.238876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:12:17.167339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383437+0000","last_change":"2024-04-18T10:05:59.256210+0000","last_active":"2024-04-18T10:06:00.383437+0000","last_peered":"2024-04-18T10:06:00.383437+0000","last_clean":"2024-04-18T10:06:00.383437+0000","last_became_active":"2024-04-18T10:05:59.255975+0000","last_became_peered":"2024-04-18T10:05:59.255975+0000","last_unstale":"2024-04-18T10:06:00.383437+0000","last_undegraded":"2024-04-18T10:06:00.383437+0000","last_fullsized":"2024-04-18T10:06:00.383437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:49:34.764977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242471+0000","last_change":"2024-04-18T10:05:59.272589+0000","last_active":"2024-04-18T10:06:00.242471+0000","last_peered":"2024-04-18T10:06:00.242471+0000","last_clean":"2024-04-18T10:06:00.242471+0000","last_became_active":"2024-04-18T10:05:59.271883+0000","last_became_peered":"2024-04-18T10:05:59.271883+0000","last_unstale":"2024-04-18T10:06:00.242471+0000","last_undegraded":"2024-04-18T10:06:00.242471+0000","last_fullsized":"2024-04-18T10:06:00.242471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:24:15.929261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243230+0000","last_change":"2024-04-18T10:05:59.261146+0000","last_active":"2024-04-18T10:06:00.243230+0000","last_peered":"2024-04-18T10:06:00.243230+0000","last_clean":"2024-04-18T10:06:00.243230+0000","last_became_active":"2024-04-18T10:05:59.260455+0000","last_became_peered":"2024-04-18T10:05:59.260455+0000","last_unstale":"2024-04-18T10:06:00.243230+0000","last_undegraded":"2024-04-18T10:06:00.243230+0000","last_fullsized":"2024-04-18T10:06:00.243230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.244095+0000","last_change":"2024-04-18T10:05:59.260958+0000","last_active":"2024-04-18T10:06:00.244095+0000","last_peered":"2024-04-18T10:06:00.244095+0000","last_clean":"2024-04-18T10:06:00.244095+0000","last_became_active":"2024-04-18T10:05:59.260251+0000","last_became_peered":"2024-04-18T10:05:59.260251+0000","last_unstale":"2024-04-18T10:06:00.244095+0000","last_undegraded":"2024-04-18T10:06:00.244095+0000","last_fullsized":"2024-04-18T10:06:00.244095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384238+0000","last_change":"2024-04-18T10:05:59.238407+0000","last_active":"2024-04-18T10:06:00.384238+0000","last_peered":"2024-04-18T10:06:00.384238+0000","last_clean":"2024-04-18T10:06:00.384238+0000","last_became_active":"2024-04-18T10:05:59.236530+0000","last_became_peered":"2024-04-18T10:05:59.236530+0000","last_unstale":"2024-04-18T10:06:00.384238+0000","last_undegraded":"2024-04-18T10:06:00.384238+0000","last_fullsized":"2024-04-18T10:06:00.384238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:51.419833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384926+0000","last_change":"2024-04-18T10:05:59.243032+0000","last_active":"2024-04-18T10:06:00.384926+0000","last_peered":"2024-04-18T10:06:00.384926+0000","last_clean":"2024-04-18T10:06:00.384926+0000","last_became_active":"2024-04-18T10:05:59.242716+0000","last_became_peered":"2024-04-18T10:05:59.242716+0000","last_unstale":"2024-04-18T10:06:00.384926+0000","last_undegraded":"2024-04-18T10:06:00.384926+0000","last_fullsized":"2024-04-18T10:06:00.384926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384052+0000","last_change":"2024-04-18T10:05:59.240754+0000","last_active":"2024-04-18T10:06:00.384052+0000","last_peered":"2024-04-18T10:06:00.384052+0000","last_clean":"2024-04-18T10:06:00.384052+0000","last_became_active":"2024-04-18T10:05:59.239992+0000","last_became_peered":"2024-04-18T10:05:59.239992+0000","last_unstale":"2024-04-18T10:06:00.384052+0000","last_undegraded":"2024-04-18T10:06:00.384052+0000","last_fullsized":"2024-04-18T10:06:00.384052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232972+0000","last_change":"2024-04-18T10:05:59.224563+0000","last_active":"2024-04-18T10:06:00.232972+0000","last_peered":"2024-04-18T10:06:00.232972+0000","last_clean":"2024-04-18T10:06:00.232972+0000","last_became_active":"2024-04-18T10:05:59.224115+0000","last_became_peered":"2024-04-18T10:05:59.224115+0000","last_unstale":"2024-04-18T10:06:00.232972+0000","last_undegraded":"2024-04-18T10:06:00.232972+0000","last_fullsized":"2024-04-18T10:06:00.232972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238461+0000","last_change":"2024-04-18T10:05:59.232777+0000","last_active":"2024-04-18T10:06:00.238461+0000","last_peered":"2024-04-18T10:06:00.238461+0000","last_clean":"2024-04-18T10:06:00.238461+0000","last_became_active":"2024-04-18T10:05:59.231188+0000","last_became_peered":"2024-04-18T10:05:59.231188+0000","last_unstale":"2024-04-18T10:06:00.238461+0000","last_undegraded":"2024-04-18T10:06:00.238461+0000","last_fullsized":"2024-04-18T10:06:00.238461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243840+0000","last_change":"2024-04-18T10:05:59.264529+0000","last_active":"2024-04-18T10:06:00.243840+0000","last_peered":"2024-04-18T10:06:00.243840+0000","last_clean":"2024-04-18T10:06:00.243840+0000","last_became_active":"2024-04-18T10:05:59.260972+0000","last_became_peered":"2024-04-18T10:05:59.260972+0000","last_unstale":"2024-04-18T10:06:00.243840+0000","last_undegraded":"2024-04-18T10:06:00.243840+0000","last_fullsized":"2024-04-18T10:06:00.243840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:24:15.929261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385847+0000","last_change":"2024-04-18T10:05:59.262102+0000","last_active":"2024-04-18T10:06:00.385847+0000","last_peered":"2024-04-18T10:06:00.385847+0000","last_clean":"2024-04-18T10:06:00.385847+0000","last_became_active":"2024-04-18T10:05:59.261796+0000","last_became_peered":"2024-04-18T10:05:59.261796+0000","last_unstale":"2024-04-18T10:06:00.385847+0000","last_undegraded":"2024-04-18T10:06:00.385847+0000","last_fullsized":"2024-04-18T10:06:00.385847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:51.419833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241173+0000","last_change":"2024-04-18T10:05:59.256981+0000","last_active":"2024-04-18T10:06:00.241173+0000","last_peered":"2024-04-18T10:06:00.241173+0000","last_clean":"2024-04-18T10:06:00.241173+0000","last_became_active":"2024-04-18T10:05:59.252309+0000","last_became_peered":"2024-04-18T10:05:59.252309+0000","last_unstale":"2024-04-18T10:06:00.241173+0000","last_undegraded":"2024-04-18T10:06:00.241173+0000","last_fullsized":"2024-04-18T10:06:00.241173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:16:23.709955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243343+0000","last_change":"2024-04-18T10:05:59.266020+0000","last_active":"2024-04-18T10:06:00.243343+0000","last_peered":"2024-04-18T10:06:00.243343+0000","last_clean":"2024-04-18T10:06:00.243343+0000","last_became_active":"2024-04-18T10:05:59.265361+0000","last_became_peered":"2024-04-18T10:05:59.265361+0000","last_unstale":"2024-04-18T10:06:00.243343+0000","last_undegraded":"2024-04-18T10:06:00.243343+0000","last_fullsized":"2024-04-18T10:06:00.243343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384730+0000","last_change":"2024-04-18T10:05:59.262190+0000","last_active":"2024-04-18T10:06:00.384730+0000","last_peered":"2024-04-18T10:06:00.384730+0000","last_clean":"2024-04-18T10:06:00.384730+0000","last_became_active":"2024-04-18T10:05:59.261866+0000","last_became_peered":"2024-04-18T10:05:59.261866+0000","last_unstale":"2024-04-18T10:06:00.384730+0000","last_undegraded":"2024-04-18T10:06:00.384730+0000","last_fullsized":"2024-04-18T10:06:00.384730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:49:34.764977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238819+0000","last_change":"2024-04-18T10:05:59.241285+0000","last_active":"2024-04-18T10:06:00.238819+0000","last_peered":"2024-04-18T10:06:00.238819+0000","last_clean":"2024-04-18T10:06:00.238819+0000","last_became_active":"2024-04-18T10:05:59.239568+0000","last_became_peered":"2024-04-18T10:05:59.239568+0000","last_unstale":"2024-04-18T10:06:00.238819+0000","last_undegraded":"2024-04-18T10:06:00.238819+0000","last_fullsized":"2024-04-18T10:06:00.238819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:12:17.167339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385177+0000","last_change":"2024-04-18T10:05:59.264180+0000","last_active":"2024-04-18T10:06:00.385177+0000","last_peered":"2024-04-18T10:06:00.385177+0000","last_clean":"2024-04-18T10:06:00.385177+0000","last_became_active":"2024-04-18T10:05:59.263866+0000","last_became_peered":"2024-04-18T10:05:59.263866+0000","last_unstale":"2024-04-18T10:06:00.385177+0000","last_undegraded":"2024-04-18T10:06:00.385177+0000","last_fullsized":"2024-04-18T10:06:00.385177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238829+0000","last_change":"2024-04-18T10:05:59.246355+0000","last_active":"2024-04-18T10:06:00.238829+0000","last_peered":"2024-04-18T10:06:00.238829+0000","last_clean":"2024-04-18T10:06:00.238829+0000","last_became_active":"2024-04-18T10:05:59.239576+0000","last_became_peered":"2024-04-18T10:05:59.239576+0000","last_unstale":"2024-04-18T10:06:00.238829+0000","last_undegraded":"2024-04-18T10:06:00.238829+0000","last_fullsized":"2024-04-18T10:06:00.238829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:27:56.094004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386366+0000","last_change":"2024-04-18T10:05:59.264586+0000","last_active":"2024-04-18T10:06:00.386366+0000","last_peered":"2024-04-18T10:06:00.386366+0000","last_clean":"2024-04-18T10:06:00.386366+0000","last_became_active":"2024-04-18T10:05:59.264215+0000","last_became_peered":"2024-04-18T10:05:59.264215+0000","last_unstale":"2024-04-18T10:06:00.386366+0000","last_undegraded":"2024-04-18T10:06:00.386366+0000","last_fullsized":"2024-04-18T10:06:00.386366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:27:56.094004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242998+0000","last_change":"2024-04-18T10:05:59.243781+0000","last_active":"2024-04-18T10:06:00.242998+0000","last_peered":"2024-04-18T10:06:00.242998+0000","last_clean":"2024-04-18T10:06:00.242998+0000","last_became_active":"2024-04-18T10:05:59.243092+0000","last_became_peered":"2024-04-18T10:05:59.243092+0000","last_unstale":"2024-04-18T10:06:00.242998+0000","last_undegraded":"2024-04-18T10:06:00.242998+0000","last_fullsized":"2024-04-18T10:06:00.242998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:49:34.764977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232294+0000","last_change":"2024-04-18T10:05:59.234128+0000","last_active":"2024-04-18T10:06:00.232294+0000","last_peered":"2024-04-18T10:06:00.232294+0000","last_clean":"2024-04-18T10:06:00.232294+0000","last_became_active":"2024-04-18T10:05:59.233742+0000","last_became_peered":"2024-04-18T10:05:59.233742+0000","last_unstale":"2024-04-18T10:06:00.232294+0000","last_undegraded":"2024-04-18T10:06:00.232294+0000","last_fullsized":"2024-04-18T10:06:00.232294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386224+0000","last_change":"2024-04-18T10:05:59.264826+0000","last_active":"2024-04-18T10:06:00.386224+0000","last_peered":"2024-04-18T10:06:00.386224+0000","last_clean":"2024-04-18T10:06:00.386224+0000","last_became_active":"2024-04-18T10:05:59.263655+0000","last_became_peered":"2024-04-18T10:05:59.263655+0000","last_unstale":"2024-04-18T10:06:00.386224+0000","last_undegraded":"2024-04-18T10:06:00.386224+0000","last_fullsized":"2024-04-18T10:06:00.386224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:12:17.167339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383983+0000","last_change":"2024-04-18T10:05:59.235032+0000","last_active":"2024-04-18T10:06:00.383983+0000","last_peered":"2024-04-18T10:06:00.383983+0000","last_clean":"2024-04-18T10:06:00.383983+0000","last_became_active":"2024-04-18T10:05:59.234660+0000","last_became_peered":"2024-04-18T10:05:59.234660+0000","last_unstale":"2024-04-18T10:06:00.383983+0000","last_undegraded":"2024-04-18T10:06:00.383983+0000","last_fullsized":"2024-04-18T10:06:00.383983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239182+0000","last_change":"2024-04-18T10:05:59.241915+0000","last_active":"2024-04-18T10:06:00.239182+0000","last_peered":"2024-04-18T10:06:00.239182+0000","last_clean":"2024-04-18T10:06:00.239182+0000","last_became_active":"2024-04-18T10:05:59.240519+0000","last_became_peered":"2024-04-18T10:05:59.240519+0000","last_unstale":"2024-04-18T10:06:00.239182+0000","last_undegraded":"2024-04-18T10:06:00.239182+0000","last_fullsized":"2024-04-18T10:06:00.239182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:34:28.871228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385003+0000","last_change":"2024-04-18T10:05:59.246146+0000","last_active":"2024-04-18T10:06:00.385003+0000","last_peered":"2024-04-18T10:06:00.385003+0000","last_clean":"2024-04-18T10:06:00.385003+0000","last_became_active":"2024-04-18T10:05:59.245800+0000","last_became_peered":"2024-04-18T10:05:59.245800+0000","last_unstale":"2024-04-18T10:06:00.385003+0000","last_undegraded":"2024-04-18T10:06:00.385003+0000","last_fullsized":"2024-04-18T10:06:00.385003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:22.118304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242883+0000","last_change":"2024-04-18T10:05:59.239684+0000","last_active":"2024-04-18T10:06:00.242883+0000","last_peered":"2024-04-18T10:06:00.242883+0000","last_clean":"2024-04-18T10:06:00.242883+0000","last_became_active":"2024-04-18T10:05:59.233494+0000","last_became_peered":"2024-04-18T10:05:59.233494+0000","last_unstale":"2024-04-18T10:06:00.242883+0000","last_undegraded":"2024-04-18T10:06:00.242883+0000","last_fullsized":"2024-04-18T10:06:00.242883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:23:54.569394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383451+0000","last_change":"2024-04-18T10:05:59.219492+0000","last_active":"2024-04-18T10:06:00.383451+0000","last_peered":"2024-04-18T10:06:00.383451+0000","last_clean":"2024-04-18T10:06:00.383451+0000","last_became_active":"2024-04-18T10:05:59.218737+0000","last_became_peered":"2024-04-18T10:05:59.218737+0000","last_unstale":"2024-04-18T10:06:00.383451+0000","last_undegraded":"2024-04-18T10:06:00.383451+0000","last_fullsized":"2024-04-18T10:06:00.383451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:33.010147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386355+0000","last_change":"2024-04-18T10:05:58.253056+0000","last_active":"2024-04-18T10:06:00.386355+0000","last_peered":"2024-04-18T10:06:00.386355+0000","last_clean":"2024-04-18T10:06:00.386355+0000","last_became_active":"2024-04-18T10:05:58.252737+0000","last_became_peered":"2024-04-18T10:05:58.252737+0000","last_unstale":"2024-04-18T10:06:00.386355+0000","last_undegraded":"2024-04-18T10:06:00.386355+0000","last_fullsized":"2024-04-18T10:06:00.386355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384430+0000","last_change":"2024-04-18T10:05:59.263750+0000","last_active":"2024-04-18T10:06:00.384430+0000","last_peered":"2024-04-18T10:06:00.384430+0000","last_clean":"2024-04-18T10:06:00.384430+0000","last_became_active":"2024-04-18T10:05:59.263538+0000","last_became_peered":"2024-04-18T10:05:59.263538+0000","last_unstale":"2024-04-18T10:06:00.384430+0000","last_undegraded":"2024-04-18T10:06:00.384430+0000","last_fullsized":"2024-04-18T10:06:00.384430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243880+0000","last_change":"2024-04-18T10:05:58.243026+0000","last_active":"2024-04-18T10:06:00.243880+0000","last_peered":"2024-04-18T10:06:00.243880+0000","last_clean":"2024-04-18T10:06:00.243880+0000","last_became_active":"2024-04-18T10:05:58.240506+0000","last_became_peered":"2024-04-18T10:05:58.240506+0000","last_unstale":"2024-04-18T10:06:00.243880+0000","last_undegraded":"2024-04-18T10:06:00.243880+0000","last_fullsized":"2024-04-18T10:06:00.243880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385279+0000","last_change":"2024-04-18T10:05:58.243128+0000","last_active":"2024-04-18T10:06:00.385279+0000","last_peered":"2024-04-18T10:06:00.385279+0000","last_clean":"2024-04-18T10:06:00.385279+0000","last_became_active":"2024-04-18T10:05:58.242694+0000","last_became_peered":"2024-04-18T10:05:58.242694+0000","last_unstale":"2024-04-18T10:06:00.385279+0000","last_undegraded":"2024-04-18T10:06:00.385279+0000","last_fullsized":"2024-04-18T10:06:00.385279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:52:26.587547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383890+0000","last_change":"2024-04-18T10:05:59.232704+0000","last_active":"2024-04-18T10:06:00.383890+0000","last_peered":"2024-04-18T10:06:00.383890+0000","last_clean":"2024-04-18T10:06:00.383890+0000","last_became_active":"2024-04-18T10:05:59.231762+0000","last_became_peered":"2024-04-18T10:05:59.231762+0000","last_unstale":"2024-04-18T10:06:00.383890+0000","last_undegraded":"2024-04-18T10:06:00.383890+0000","last_fullsized":"2024-04-18T10:06:00.383890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239055+0000","last_change":"2024-04-18T10:05:58.231745+0000","last_active":"2024-04-18T10:06:00.239055+0000","last_peered":"2024-04-18T10:06:00.239055+0000","last_clean":"2024-04-18T10:06:00.239055+0000","last_became_active":"2024-04-18T10:05:58.230994+0000","last_became_peered":"2024-04-18T10:05:58.230994+0000","last_unstale":"2024-04-18T10:06:00.239055+0000","last_undegraded":"2024-04-18T10:06:00.239055+0000","last_fullsized":"2024-04-18T10:06:00.239055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232217+0000","last_change":"2024-04-18T10:05:59.232550+0000","last_active":"2024-04-18T10:06:00.232217+0000","last_peered":"2024-04-18T10:06:00.232217+0000","last_clean":"2024-04-18T10:06:00.232217+0000","last_became_active":"2024-04-18T10:05:59.231955+0000","last_became_peered":"2024-04-18T10:05:59.231955+0000","last_unstale":"2024-04-18T10:06:00.232217+0000","last_undegraded":"2024-04-18T10:06:00.232217+0000","last_fullsized":"2024-04-18T10:06:00.232217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243516+0000","last_change":"2024-04-18T10:05:58.244903+0000","last_active":"2024-04-18T10:06:00.243516+0000","last_peered":"2024-04-18T10:06:00.243516+0000","last_clean":"2024-04-18T10:06:00.243516+0000","last_became_active":"2024-04-18T10:05:58.244287+0000","last_became_peered":"2024-04-18T10:05:58.244287+0000","last_unstale":"2024-04-18T10:06:00.243516+0000","last_undegraded":"2024-04-18T10:06:00.243516+0000","last_fullsized":"2024-04-18T10:06:00.243516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240811+0000","last_change":"2024-04-18T10:05:59.238308+0000","last_active":"2024-04-18T10:06:00.240811+0000","last_peered":"2024-04-18T10:06:00.240811+0000","last_clean":"2024-04-18T10:06:00.240811+0000","last_became_active":"2024-04-18T10:05:59.237422+0000","last_became_peered":"2024-04-18T10:05:59.237422+0000","last_unstale":"2024-04-18T10:06:00.240811+0000","last_undegraded":"2024-04-18T10:06:00.240811+0000","last_fullsized":"2024-04-18T10:06:00.240811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:28:59.797065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240802+0000","last_change":"2024-04-18T10:05:58.245925+0000","last_active":"2024-04-18T10:06:00.240802+0000","last_peered":"2024-04-18T10:06:00.240802+0000","last_clean":"2024-04-18T10:06:00.240802+0000","last_became_active":"2024-04-18T10:05:58.244713+0000","last_became_peered":"2024-04-18T10:05:58.244713+0000","last_unstale":"2024-04-18T10:06:00.240802+0000","last_undegraded":"2024-04-18T10:06:00.240802+0000","last_fullsized":"2024-04-18T10:06:00.240802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384230+0000","last_change":"2024-04-18T10:05:59.252961+0000","last_active":"2024-04-18T10:06:00.384230+0000","last_peered":"2024-04-18T10:06:00.384230+0000","last_clean":"2024-04-18T10:06:00.384230+0000","last_became_active":"2024-04-18T10:05:59.252672+0000","last_became_peered":"2024-04-18T10:05:59.252672+0000","last_unstale":"2024-04-18T10:06:00.384230+0000","last_undegraded":"2024-04-18T10:06:00.384230+0000","last_fullsized":"2024-04-18T10:06:00.384230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232406+0000","last_change":"2024-04-18T10:05:58.235997+0000","last_active":"2024-04-18T10:06:00.232406+0000","last_peered":"2024-04-18T10:06:00.232406+0000","last_clean":"2024-04-18T10:06:00.232406+0000","last_became_active":"2024-04-18T10:05:58.235330+0000","last_became_peered":"2024-04-18T10:05:58.235330+0000","last_unstale":"2024-04-18T10:06:00.232406+0000","last_undegraded":"2024-04-18T10:06:00.232406+0000","last_fullsized":"2024-04-18T10:06:00.232406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:21.107270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384433+0000","last_change":"2024-04-18T10:05:59.257176+0000","last_active":"2024-04-18T10:06:00.384433+0000","last_peered":"2024-04-18T10:06:00.384433+0000","last_clean":"2024-04-18T10:06:00.384433+0000","last_became_active":"2024-04-18T10:05:59.256452+0000","last_became_peered":"2024-04-18T10:05:59.256452+0000","last_unstale":"2024-04-18T10:06:00.384433+0000","last_undegraded":"2024-04-18T10:06:00.384433+0000","last_fullsized":"2024-04-18T10:06:00.384433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:04:27.662085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243463+0000","last_change":"2024-04-18T10:05:58.239207+0000","last_active":"2024-04-18T10:06:00.243463+0000","last_peered":"2024-04-18T10:06:00.243463+0000","last_clean":"2024-04-18T10:06:00.243463+0000","last_became_active":"2024-04-18T10:05:58.238349+0000","last_became_peered":"2024-04-18T10:05:58.238349+0000","last_unstale":"2024-04-18T10:06:00.243463+0000","last_undegraded":"2024-04-18T10:06:00.243463+0000","last_fullsized":"2024-04-18T10:06:00.243463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239372+0000","last_change":"2024-04-18T10:05:59.254087+0000","last_active":"2024-04-18T10:06:00.239372+0000","last_peered":"2024-04-18T10:06:00.239372+0000","last_clean":"2024-04-18T10:06:00.239372+0000","last_became_active":"2024-04-18T10:05:59.253296+0000","last_became_peered":"2024-04-18T10:05:59.253296+0000","last_unstale":"2024-04-18T10:06:00.239372+0000","last_undegraded":"2024-04-18T10:06:00.239372+0000","last_fullsized":"2024-04-18T10:06:00.239372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:27:56.094004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384701+0000","last_change":"2024-04-18T10:05:58.236375+0000","last_active":"2024-04-18T10:06:00.384701+0000","last_peered":"2024-04-18T10:06:00.384701+0000","last_clean":"2024-04-18T10:06:00.384701+0000","last_became_active":"2024-04-18T10:05:58.236066+0000","last_became_peered":"2024-04-18T10:05:58.236066+0000","last_unstale":"2024-04-18T10:06:00.384701+0000","last_undegraded":"2024-04-18T10:06:00.384701+0000","last_fullsized":"2024-04-18T10:06:00.384701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:52:26.587547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383621+0000","last_change":"2024-04-18T10:05:59.262667+0000","last_active":"2024-04-18T10:06:00.383621+0000","last_peered":"2024-04-18T10:06:00.383621+0000","last_clean":"2024-04-18T10:06:00.383621+0000","last_became_active":"2024-04-18T10:05:59.262365+0000","last_became_peered":"2024-04-18T10:05:59.262365+0000","last_unstale":"2024-04-18T10:06:00.383621+0000","last_undegraded":"2024-04-18T10:06:00.383621+0000","last_fullsized":"2024-04-18T10:06:00.383621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:23:54.569394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239431+0000","last_change":"2024-04-18T10:05:58.243114+0000","last_active":"2024-04-18T10:06:00.239431+0000","last_peered":"2024-04-18T10:06:00.239431+0000","last_clean":"2024-04-18T10:06:00.239431+0000","last_became_active":"2024-04-18T10:05:58.242102+0000","last_became_peered":"2024-04-18T10:05:58.242102+0000","last_unstale":"2024-04-18T10:06:00.239431+0000","last_undegraded":"2024-04-18T10:06:00.239431+0000","last_fullsized":"2024-04-18T10:06:00.239431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239878+0000","last_change":"2024-04-18T10:05:59.274245+0000","last_active":"2024-04-18T10:06:00.239878+0000","last_peered":"2024-04-18T10:06:00.239878+0000","last_clean":"2024-04-18T10:06:00.239878+0000","last_became_active":"2024-04-18T10:05:59.273707+0000","last_became_peered":"2024-04-18T10:05:59.273707+0000","last_unstale":"2024-04-18T10:06:00.239878+0000","last_undegraded":"2024-04-18T10:06:00.239878+0000","last_fullsized":"2024-04-18T10:06:00.239878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:49.354229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243436+0000","last_change":"2024-04-18T10:05:58.242951+0000","last_active":"2024-04-18T10:06:00.243436+0000","last_peered":"2024-04-18T10:06:00.243436+0000","last_clean":"2024-04-18T10:06:00.243436+0000","last_became_active":"2024-04-18T10:05:58.241438+0000","last_became_peered":"2024-04-18T10:05:58.241438+0000","last_unstale":"2024-04-18T10:06:00.243436+0000","last_undegraded":"2024-04-18T10:06:00.243436+0000","last_fullsized":"2024-04-18T10:06:00.243436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243154+0000","last_change":"2024-04-18T10:05:59.233588+0000","last_active":"2024-04-18T10:06:00.243154+0000","last_peered":"2024-04-18T10:06:00.243154+0000","last_clean":"2024-04-18T10:06:00.243154+0000","last_became_active":"2024-04-18T10:05:59.232831+0000","last_became_peered":"2024-04-18T10:05:59.232831+0000","last_unstale":"2024-04-18T10:06:00.243154+0000","last_undegraded":"2024-04-18T10:06:00.243154+0000","last_fullsized":"2024-04-18T10:06:00.243154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384620+0000","last_change":"2024-04-18T10:05:58.223796+0000","last_active":"2024-04-18T10:06:00.384620+0000","last_peered":"2024-04-18T10:06:00.384620+0000","last_clean":"2024-04-18T10:06:00.384620+0000","last_became_active":"2024-04-18T10:05:58.223248+0000","last_became_peered":"2024-04-18T10:05:58.223248+0000","last_unstale":"2024-04-18T10:06:00.384620+0000","last_undegraded":"2024-04-18T10:06:00.384620+0000","last_fullsized":"2024-04-18T10:06:00.384620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:52:26.587547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240028+0000","last_change":"2024-04-18T10:05:59.277176+0000","last_active":"2024-04-18T10:06:00.240028+0000","last_peered":"2024-04-18T10:06:00.240028+0000","last_clean":"2024-04-18T10:06:00.240028+0000","last_became_active":"2024-04-18T10:05:59.276386+0000","last_became_peered":"2024-04-18T10:05:59.276386+0000","last_unstale":"2024-04-18T10:06:00.240028+0000","last_undegraded":"2024-04-18T10:06:00.240028+0000","last_fullsized":"2024-04-18T10:06:00.240028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:28:59.797065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384638+0000","last_change":"2024-04-18T10:05:58.233338+0000","last_active":"2024-04-18T10:06:00.384638+0000","last_peered":"2024-04-18T10:06:00.384638+0000","last_clean":"2024-04-18T10:06:00.384638+0000","last_became_active":"2024-04-18T10:05:58.231906+0000","last_became_peered":"2024-04-18T10:05:58.231906+0000","last_unstale":"2024-04-18T10:06:00.384638+0000","last_undegraded":"2024-04-18T10:06:00.384638+0000","last_fullsized":"2024-04-18T10:06:00.384638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:06:57.988393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241935+0000","last_change":"2024-04-18T10:05:59.275977+0000","last_active":"2024-04-18T10:06:00.241935+0000","last_peered":"2024-04-18T10:06:00.241935+0000","last_clean":"2024-04-18T10:06:00.241935+0000","last_became_active":"2024-04-18T10:05:59.275386+0000","last_became_peered":"2024-04-18T10:05:59.275386+0000","last_unstale":"2024-04-18T10:06:00.241935+0000","last_undegraded":"2024-04-18T10:06:00.241935+0000","last_fullsized":"2024-04-18T10:06:00.241935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:58:16.367066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386132+0000","last_change":"2024-04-18T10:05:58.229897+0000","last_active":"2024-04-18T10:06:00.386132+0000","last_peered":"2024-04-18T10:06:00.386132+0000","last_clean":"2024-04-18T10:06:00.386132+0000","last_became_active":"2024-04-18T10:05:58.229451+0000","last_became_peered":"2024-04-18T10:05:58.229451+0000","last_unstale":"2024-04-18T10:06:00.386132+0000","last_undegraded":"2024-04-18T10:06:00.386132+0000","last_fullsized":"2024-04-18T10:06:00.386132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242763+0000","last_change":"2024-04-18T10:05:59.276815+0000","last_active":"2024-04-18T10:06:00.242763+0000","last_peered":"2024-04-18T10:06:00.242763+0000","last_clean":"2024-04-18T10:06:00.242763+0000","last_became_active":"2024-04-18T10:05:59.276272+0000","last_became_peered":"2024-04-18T10:05:59.276272+0000","last_unstale":"2024-04-18T10:06:00.242763+0000","last_undegraded":"2024-04-18T10:06:00.242763+0000","last_fullsized":"2024-04-18T10:06:00.242763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:44:45.775310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232479+0000","last_change":"2024-04-18T10:05:58.230942+0000","last_active":"2024-04-18T10:06:00.232479+0000","last_peered":"2024-04-18T10:06:00.232479+0000","last_clean":"2024-04-18T10:06:00.232479+0000","last_became_active":"2024-04-18T10:05:58.230373+0000","last_became_peered":"2024-04-18T10:05:58.230373+0000","last_unstale":"2024-04-18T10:06:00.232479+0000","last_undegraded":"2024-04-18T10:06:00.232479+0000","last_fullsized":"2024-04-18T10:06:00.232479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:50.408798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241359+0000","last_change":"2024-04-18T10:05:59.234199+0000","last_active":"2024-04-18T10:06:00.241359+0000","last_peered":"2024-04-18T10:06:00.241359+0000","last_clean":"2024-04-18T10:06:00.241359+0000","last_became_active":"2024-04-18T10:05:59.233412+0000","last_became_peered":"2024-04-18T10:05:59.233412+0000","last_unstale":"2024-04-18T10:06:00.241359+0000","last_undegraded":"2024-04-18T10:06:00.241359+0000","last_fullsized":"2024-04-18T10:06:00.241359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:16:23.709955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385242+0000","last_change":"2024-04-18T10:05:58.222111+0000","last_active":"2024-04-18T10:06:00.385242+0000","last_peered":"2024-04-18T10:06:00.385242+0000","last_clean":"2024-04-18T10:06:00.385242+0000","last_became_active":"2024-04-18T10:05:58.221783+0000","last_became_peered":"2024-04-18T10:05:58.221783+0000","last_unstale":"2024-04-18T10:06:00.385242+0000","last_undegraded":"2024-04-18T10:06:00.385242+0000","last_fullsized":"2024-04-18T10:06:00.385242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240501+0000","last_change":"2024-04-18T10:05:59.261884+0000","last_active":"2024-04-18T10:06:00.240501+0000","last_peered":"2024-04-18T10:06:00.240501+0000","last_clean":"2024-04-18T10:06:00.240501+0000","last_became_active":"2024-04-18T10:05:59.261147+0000","last_became_peered":"2024-04-18T10:05:59.261147+0000","last_unstale":"2024-04-18T10:06:00.240501+0000","last_undegraded":"2024-04-18T10:06:00.240501+0000","last_fullsized":"2024-04-18T10:06:00.240501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:44:42.861177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238296+0000","last_change":"2024-04-18T10:05:49.183850+0000","last_active":"2024-04-18T10:06:00.238296+0000","last_peered":"2024-04-18T10:06:00.238296+0000","last_clean":"2024-04-18T10:06:00.238296+0000","last_became_active":"2024-04-18T10:05:49.181471+0000","last_became_peered":"2024-04-18T10:05:49.181471+0000","last_unstale":"2024-04-18T10:06:00.238296+0000","last_undegraded":"2024-04-18T10:06:00.238296+0000","last_fullsized":"2024-04-18T10:06:00.238296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_clean_scrub_stamp":"2024-04-18T10:05:47.511486+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:06:51.214547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239978+0000","last_change":"2024-04-18T10:05:58.228257+0000","last_active":"2024-04-18T10:06:00.239978+0000","last_peered":"2024-04-18T10:06:00.239978+0000","last_clean":"2024-04-18T10:06:00.239978+0000","last_became_active":"2024-04-18T10:05:58.224594+0000","last_became_peered":"2024-04-18T10:05:58.224594+0000","last_unstale":"2024-04-18T10:06:00.239978+0000","last_undegraded":"2024-04-18T10:06:00.239978+0000","last_fullsized":"2024-04-18T10:06:00.239978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385522+0000","last_change":"2024-04-18T10:05:59.256229+0000","last_active":"2024-04-18T10:06:00.385522+0000","last_peered":"2024-04-18T10:06:00.385522+0000","last_clean":"2024-04-18T10:06:00.385522+0000","last_became_active":"2024-04-18T10:05:59.255894+0000","last_became_peered":"2024-04-18T10:05:59.255894+0000","last_unstale":"2024-04-18T10:06:00.385522+0000","last_undegraded":"2024-04-18T10:06:00.385522+0000","last_fullsized":"2024-04-18T10:06:00.385522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238224+0000","last_change":"2024-04-18T10:05:58.254200+0000","last_active":"2024-04-18T10:06:00.238224+0000","last_peered":"2024-04-18T10:06:00.238224+0000","last_clean":"2024-04-18T10:06:00.238224+0000","last_became_active":"2024-04-18T10:05:58.253365+0000","last_became_peered":"2024-04-18T10:05:58.253365+0000","last_unstale":"2024-04-18T10:06:00.238224+0000","last_undegraded":"2024-04-18T10:06:00.238224+0000","last_fullsized":"2024-04-18T10:06:00.238224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383217+0000","last_change":"2024-04-18T10:05:59.264229+0000","last_active":"2024-04-18T10:06:00.383217+0000","last_peered":"2024-04-18T10:06:00.383217+0000","last_clean":"2024-04-18T10:06:00.383217+0000","last_became_active":"2024-04-18T10:05:59.263577+0000","last_became_peered":"2024-04-18T10:05:59.263577+0000","last_unstale":"2024-04-18T10:06:00.383217+0000","last_undegraded":"2024-04-18T10:06:00.383217+0000","last_fullsized":"2024-04-18T10:06:00.383217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:37:20.126666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385257+0000","last_change":"2024-04-18T10:05:58.235466+0000","last_active":"2024-04-18T10:06:00.385257+0000","last_peered":"2024-04-18T10:06:00.385257+0000","last_clean":"2024-04-18T10:06:00.385257+0000","last_became_active":"2024-04-18T10:05:58.234823+0000","last_became_peered":"2024-04-18T10:05:58.234823+0000","last_unstale":"2024-04-18T10:06:00.385257+0000","last_undegraded":"2024-04-18T10:06:00.385257+0000","last_fullsized":"2024-04-18T10:06:00.385257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232451+0000","last_change":"2024-04-18T10:05:59.230409+0000","last_active":"2024-04-18T10:06:00.232451+0000","last_peered":"2024-04-18T10:06:00.232451+0000","last_clean":"2024-04-18T10:06:00.232451+0000","last_became_active":"2024-04-18T10:05:59.228899+0000","last_became_peered":"2024-04-18T10:05:59.228899+0000","last_unstale":"2024-04-18T10:06:00.232451+0000","last_undegraded":"2024-04-18T10:06:00.232451+0000","last_fullsized":"2024-04-18T10:06:00.232451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239789+0000","last_change":"2024-04-18T10:05:58.232028+0000","last_active":"2024-04-18T10:06:00.239789+0000","last_peered":"2024-04-18T10:06:00.239789+0000","last_clean":"2024-04-18T10:06:00.239789+0000","last_became_active":"2024-04-18T10:05:58.229311+0000","last_became_peered":"2024-04-18T10:05:58.229311+0000","last_unstale":"2024-04-18T10:06:00.239789+0000","last_undegraded":"2024-04-18T10:06:00.239789+0000","last_fullsized":"2024-04-18T10:06:00.239789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383872+0000","last_change":"2024-04-18T10:05:59.248629+0000","last_active":"2024-04-18T10:06:00.383872+0000","last_peered":"2024-04-18T10:06:00.383872+0000","last_clean":"2024-04-18T10:06:00.383872+0000","last_became_active":"2024-04-18T10:05:59.247619+0000","last_became_peered":"2024-04-18T10:05:59.247619+0000","last_unstale":"2024-04-18T10:06:00.383872+0000","last_undegraded":"2024-04-18T10:06:00.383872+0000","last_fullsized":"2024-04-18T10:06:00.383872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:35.965832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232588+0000","last_change":"2024-04-18T10:05:58.224570+0000","last_active":"2024-04-18T10:06:00.232588+0000","last_peered":"2024-04-18T10:06:00.232588+0000","last_clean":"2024-04-18T10:06:00.232588+0000","last_became_active":"2024-04-18T10:05:58.224216+0000","last_became_peered":"2024-04-18T10:05:58.224216+0000","last_unstale":"2024-04-18T10:06:00.232588+0000","last_undegraded":"2024-04-18T10:06:00.232588+0000","last_fullsized":"2024-04-18T10:06:00.232588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242688+0000","last_change":"2024-04-18T10:05:59.246784+0000","last_active":"2024-04-18T10:06:00.242688+0000","last_peered":"2024-04-18T10:06:00.242688+0000","last_clean":"2024-04-18T10:06:00.242688+0000","last_became_active":"2024-04-18T10:05:59.243518+0000","last_became_peered":"2024-04-18T10:05:59.243518+0000","last_unstale":"2024-04-18T10:06:00.242688+0000","last_undegraded":"2024-04-18T10:06:00.242688+0000","last_fullsized":"2024-04-18T10:06:00.242688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:45:43.670227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241271+0000","last_change":"2024-04-18T10:05:58.258428+0000","last_active":"2024-04-18T10:06:00.241271+0000","last_peered":"2024-04-18T10:06:00.241271+0000","last_clean":"2024-04-18T10:06:00.241271+0000","last_became_active":"2024-04-18T10:05:58.257648+0000","last_became_peered":"2024-04-18T10:05:58.257648+0000","last_unstale":"2024-04-18T10:06:00.241271+0000","last_undegraded":"2024-04-18T10:06:00.241271+0000","last_fullsized":"2024-04-18T10:06:00.241271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:15:05.853314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232543+0000","last_change":"2024-04-18T10:05:59.229368+0000","last_active":"2024-04-18T10:06:00.232543+0000","last_peered":"2024-04-18T10:06:00.232543+0000","last_clean":"2024-04-18T10:06:00.232543+0000","last_became_active":"2024-04-18T10:05:59.228912+0000","last_became_peered":"2024-04-18T10:05:59.228912+0000","last_unstale":"2024-04-18T10:06:00.232543+0000","last_undegraded":"2024-04-18T10:06:00.232543+0000","last_fullsized":"2024-04-18T10:06:00.232543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241350+0000","last_change":"2024-04-18T10:05:58.243743+0000","last_active":"2024-04-18T10:06:00.241350+0000","last_peered":"2024-04-18T10:06:00.241350+0000","last_clean":"2024-04-18T10:06:00.241350+0000","last_became_active":"2024-04-18T10:05:58.243448+0000","last_became_peered":"2024-04-18T10:05:58.243448+0000","last_unstale":"2024-04-18T10:06:00.241350+0000","last_undegraded":"2024-04-18T10:06:00.241350+0000","last_fullsized":"2024-04-18T10:06:00.241350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:04:26.651051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386763+0000","last_change":"2024-04-18T10:05:59.262383+0000","last_active":"2024-04-18T10:06:00.386763+0000","last_peered":"2024-04-18T10:06:00.386763+0000","last_clean":"2024-04-18T10:06:00.386763+0000","last_became_active":"2024-04-18T10:05:59.262039+0000","last_became_peered":"2024-04-18T10:05:59.262039+0000","last_unstale":"2024-04-18T10:06:00.386763+0000","last_undegraded":"2024-04-18T10:06:00.386763+0000","last_fullsized":"2024-04-18T10:06:00.386763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:49.354229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243760+0000","last_change":"2024-04-18T10:05:58.251779+0000","last_active":"2024-04-18T10:06:00.243760+0000","last_peered":"2024-04-18T10:06:00.243760+0000","last_clean":"2024-04-18T10:06:00.243760+0000","last_became_active":"2024-04-18T10:05:58.251062+0000","last_became_peered":"2024-04-18T10:05:58.251062+0000","last_unstale":"2024-04-18T10:06:00.243760+0000","last_undegraded":"2024-04-18T10:06:00.243760+0000","last_fullsized":"2024-04-18T10:06:00.243760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384029+0000","last_change":"2024-04-18T10:05:59.240684+0000","last_active":"2024-04-18T10:06:00.384029+0000","last_peered":"2024-04-18T10:06:00.384029+0000","last_clean":"2024-04-18T10:06:00.384029+0000","last_became_active":"2024-04-18T10:05:59.240231+0000","last_became_peered":"2024-04-18T10:05:59.240231+0000","last_unstale":"2024-04-18T10:06:00.384029+0000","last_undegraded":"2024-04-18T10:06:00.384029+0000","last_fullsized":"2024-04-18T10:06:00.384029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238119+0000","last_change":"2024-04-18T10:05:58.254872+0000","last_active":"2024-04-18T10:06:00.238119+0000","last_peered":"2024-04-18T10:06:00.238119+0000","last_clean":"2024-04-18T10:06:00.238119+0000","last_became_active":"2024-04-18T10:05:58.252439+0000","last_became_peered":"2024-04-18T10:05:58.252439+0000","last_unstale":"2024-04-18T10:06:00.238119+0000","last_undegraded":"2024-04-18T10:06:00.238119+0000","last_fullsized":"2024-04-18T10:06:00.238119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384090+0000","last_change":"2024-04-18T10:05:59.254583+0000","last_active":"2024-04-18T10:06:00.384090+0000","last_peered":"2024-04-18T10:06:00.384090+0000","last_clean":"2024-04-18T10:06:00.384090+0000","last_became_active":"2024-04-18T10:05:59.253836+0000","last_became_peered":"2024-04-18T10:05:59.253836+0000","last_unstale":"2024-04-18T10:06:00.384090+0000","last_undegraded":"2024-04-18T10:06:00.384090+0000","last_fullsized":"2024-04-18T10:06:00.384090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:22.118304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232357+0000","last_change":"2024-04-18T10:05:58.248261+0000","last_active":"2024-04-18T10:06:00.232357+0000","last_peered":"2024-04-18T10:06:00.232357+0000","last_clean":"2024-04-18T10:06:00.232357+0000","last_became_active":"2024-04-18T10:05:58.247708+0000","last_became_peered":"2024-04-18T10:05:58.247708+0000","last_unstale":"2024-04-18T10:06:00.232357+0000","last_undegraded":"2024-04-18T10:06:00.232357+0000","last_fullsized":"2024-04-18T10:06:00.232357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384524+0000","last_change":"2024-04-18T10:05:59.238345+0000","last_active":"2024-04-18T10:06:00.384524+0000","last_peered":"2024-04-18T10:06:00.384524+0000","last_clean":"2024-04-18T10:06:00.384524+0000","last_became_active":"2024-04-18T10:05:59.237905+0000","last_became_peered":"2024-04-18T10:05:59.237905+0000","last_unstale":"2024-04-18T10:06:00.384524+0000","last_undegraded":"2024-04-18T10:06:00.384524+0000","last_fullsized":"2024-04-18T10:06:00.384524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232345+0000","last_change":"2024-04-18T10:05:58.229628+0000","last_active":"2024-04-18T10:06:00.232345+0000","last_peered":"2024-04-18T10:06:00.232345+0000","last_clean":"2024-04-18T10:06:00.232345+0000","last_became_active":"2024-04-18T10:05:58.229323+0000","last_became_peered":"2024-04-18T10:05:58.229323+0000","last_unstale":"2024-04-18T10:06:00.232345+0000","last_undegraded":"2024-04-18T10:06:00.232345+0000","last_fullsized":"2024-04-18T10:06:00.232345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385760+0000","last_change":"2024-04-18T10:05:59.232981+0000","last_active":"2024-04-18T10:06:00.385760+0000","last_peered":"2024-04-18T10:06:00.385760+0000","last_clean":"2024-04-18T10:06:00.385760+0000","last_became_active":"2024-04-18T10:05:59.232662+0000","last_became_peered":"2024-04-18T10:05:59.232662+0000","last_unstale":"2024-04-18T10:06:00.385760+0000","last_undegraded":"2024-04-18T10:06:00.385760+0000","last_fullsized":"2024-04-18T10:06:00.385760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239926+0000","last_change":"2024-04-18T10:05:58.258738+0000","last_active":"2024-04-18T10:06:00.239926+0000","last_peered":"2024-04-18T10:06:00.239926+0000","last_clean":"2024-04-18T10:06:00.239926+0000","last_became_active":"2024-04-18T10:05:58.257527+0000","last_became_peered":"2024-04-18T10:05:58.257527+0000","last_unstale":"2024-04-18T10:06:00.239926+0000","last_undegraded":"2024-04-18T10:06:00.239926+0000","last_fullsized":"2024-04-18T10:06:00.239926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:21.107270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243622+0000","last_change":"2024-04-18T10:05:59.264420+0000","last_active":"2024-04-18T10:06:00.243622+0000","last_peered":"2024-04-18T10:06:00.243622+0000","last_clean":"2024-04-18T10:06:00.243622+0000","last_became_active":"2024-04-18T10:05:59.263738+0000","last_became_peered":"2024-04-18T10:05:59.263738+0000","last_unstale":"2024-04-18T10:06:00.243622+0000","last_undegraded":"2024-04-18T10:06:00.243622+0000","last_fullsized":"2024-04-18T10:06:00.243622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242313+0000","last_change":"2024-04-18T10:05:59.222327+0000","last_active":"2024-04-18T10:06:00.242313+0000","last_peered":"2024-04-18T10:06:00.242313+0000","last_clean":"2024-04-18T10:06:00.242313+0000","last_became_active":"2024-04-18T10:05:59.221623+0000","last_became_peered":"2024-04-18T10:05:59.221623+0000","last_unstale":"2024-04-18T10:06:00.242313+0000","last_undegraded":"2024-04-18T10:06:00.242313+0000","last_fullsized":"2024-04-18T10:06:00.242313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:58:16.367066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239652+0000","last_change":"2024-04-18T10:05:58.255714+0000","last_active":"2024-04-18T10:06:00.239652+0000","last_peered":"2024-04-18T10:06:00.239652+0000","last_clean":"2024-04-18T10:06:00.239652+0000","last_became_active":"2024-04-18T10:05:58.255115+0000","last_became_peered":"2024-04-18T10:05:58.255115+0000","last_unstale":"2024-04-18T10:06:00.239652+0000","last_undegraded":"2024-04-18T10:06:00.239652+0000","last_fullsized":"2024-04-18T10:06:00.239652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:50.408798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240843+0000","last_change":"2024-04-18T10:05:59.244685+0000","last_active":"2024-04-18T10:06:00.240843+0000","last_peered":"2024-04-18T10:06:00.240843+0000","last_clean":"2024-04-18T10:06:00.240843+0000","last_became_active":"2024-04-18T10:05:59.243811+0000","last_became_peered":"2024-04-18T10:05:59.243811+0000","last_unstale":"2024-04-18T10:06:00.240843+0000","last_undegraded":"2024-04-18T10:06:00.240843+0000","last_fullsized":"2024-04-18T10:06:00.240843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:44:42.861177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241014+0000","last_change":"2024-04-18T10:05:58.256922+0000","last_active":"2024-04-18T10:06:00.241014+0000","last_peered":"2024-04-18T10:06:00.241014+0000","last_clean":"2024-04-18T10:06:00.241014+0000","last_became_active":"2024-04-18T10:05:58.255255+0000","last_became_peered":"2024-04-18T10:05:58.255255+0000","last_unstale":"2024-04-18T10:06:00.241014+0000","last_undegraded":"2024-04-18T10:06:00.241014+0000","last_fullsized":"2024-04-18T10:06:00.241014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:04:26.651051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383632+0000","last_change":"2024-04-18T10:05:59.253025+0000","last_active":"2024-04-18T10:06:00.383632+0000","last_peered":"2024-04-18T10:06:00.383632+0000","last_clean":"2024-04-18T10:06:00.383632+0000","last_became_active":"2024-04-18T10:05:59.252708+0000","last_became_peered":"2024-04-18T10:05:59.252708+0000","last_unstale":"2024-04-18T10:06:00.383632+0000","last_undegraded":"2024-04-18T10:06:00.383632+0000","last_fullsized":"2024-04-18T10:06:00.383632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:17:23.117921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386955+0000","last_change":"2024-04-18T10:05:58.234122+0000","last_active":"2024-04-18T10:06:00.386955+0000","last_peered":"2024-04-18T10:06:00.386955+0000","last_clean":"2024-04-18T10:06:00.386955+0000","last_became_active":"2024-04-18T10:05:58.233320+0000","last_became_peered":"2024-04-18T10:05:58.233320+0000","last_unstale":"2024-04-18T10:06:00.386955+0000","last_undegraded":"2024-04-18T10:06:00.386955+0000","last_fullsized":"2024-04-18T10:06:00.386955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386581+0000","last_change":"2024-04-18T10:05:59.244230+0000","last_active":"2024-04-18T10:06:00.386581+0000","last_peered":"2024-04-18T10:06:00.386581+0000","last_clean":"2024-04-18T10:06:00.386581+0000","last_became_active":"2024-04-18T10:05:59.243879+0000","last_became_peered":"2024-04-18T10:05:59.243879+0000","last_unstale":"2024-04-18T10:06:00.386581+0000","last_undegraded":"2024-04-18T10:06:00.386581+0000","last_fullsized":"2024-04-18T10:06:00.386581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:34:28.871228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383948+0000","last_change":"2024-04-18T10:05:58.247996+0000","last_active":"2024-04-18T10:06:00.383948+0000","last_peered":"2024-04-18T10:06:00.383948+0000","last_clean":"2024-04-18T10:06:00.383948+0000","last_became_active":"2024-04-18T10:05:58.247612+0000","last_became_peered":"2024-04-18T10:05:58.247612+0000","last_unstale":"2024-04-18T10:06:00.383948+0000","last_undegraded":"2024-04-18T10:06:00.383948+0000","last_fullsized":"2024-04-18T10:06:00.383948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:13:49.398609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T10:06:03.323 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T10:06:03.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae3efe640 1 Processor -- start 2024-04-18T10:06:03.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae3efe640 1 -- start start 2024-04-18T10:06:03.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae3efe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdadc13ebd0 0x7fdadc0b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae3efe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdadc0b8db0 0x7fdadc0b9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae3efe640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdadc0b6280 con 0x7fdadc147f60 2024-04-18T10:06:03.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae3efe640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fdadc0b68b0 con 0x7fdadc0b8db0 2024-04-18T10:06:03.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae3efe640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fdadc0b6d40 con 0x7fdadc13ebd0 2024-04-18T10:06:03.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae1499640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdadc0b8db0 0x7fdadc0b9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae1499640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fdadc0b8db0 0x7fdadc0b9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53090/0 (socket says 172.21.15.62:53090) 2024-04-18T10:06:03.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae1499640 1 -- 172.21.15.62:0/3519139987 learned_addr learned my addr 172.21.15.62:0/3519139987 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:03.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae1499640 1 -- 172.21.15.62:0/3519139987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdadc13ebd0 msgr2=0x7fdadc0b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:03.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae1c9a640 1 --2- 172.21.15.62:0/3519139987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdadc13ebd0 0x7fdadc0b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae1499640 1 --2- 172.21.15.62:0/3519139987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdadc13ebd0 0x7fdadc0b82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae1499640 1 -- 172.21.15.62:0/3519139987 >> v1:172.21.15.145:6789/0 conn(0x7fdadc147f60 legacy=0x7fdadc1449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:03.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.486+0000 7fdae1499640 1 -- 172.21.15.62:0/3519139987 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdadc0b6540 con 0x7fdadc0b8db0 2024-04-18T10:06:03.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae1499640 1 --2- 172.21.15.62:0/3519139987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdadc0b8db0 0x7fdadc0b9a60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0b6a30 tx=0x7fdacc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4431248d5bea044d server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:03.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae0c98640 1 -- 172.21.15.62:0/3519139987 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc0b0070 con 0x7fdadc0b8db0 2024-04-18T10:06:03.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae0c98640 1 -- 172.21.15.62:0/3519139987 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdacc0ab600 con 0x7fdadc0b8db0 2024-04-18T10:06:03.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae0c98640 1 -- 172.21.15.62:0/3519139987 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdacc0ab9d0 con 0x7fdadc0b8db0 2024-04-18T10:06:03.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 -- 172.21.15.62:0/3519139987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdadc0b8db0 msgr2=0x7fdadc0b9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 --2- 172.21.15.62:0/3519139987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdadc0b8db0 0x7fdadc0b9a60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0b6a30 tx=0x7fdacc01c470 comp rx=0 tx=0).stop 2024-04-18T10:06:03.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 -- 172.21.15.62:0/3519139987 shutdown_connections 2024-04-18T10:06:03.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 --2- 172.21.15.62:0/3519139987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdadc0b8db0 0x7fdadc0b9a60 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 --2- 172.21.15.62:0/3519139987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdadc13ebd0 0x7fdadc0b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 -- 172.21.15.62:0/3519139987 >> 172.21.15.62:0/3519139987 conn(0x7fdadc09c250 msgr2=0x7fdadc09c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:03.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 -- 172.21.15.62:0/3519139987 shutdown_connections 2024-04-18T10:06:03.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 -- 172.21.15.62:0/3519139987 wait complete. 2024-04-18T10:06:03.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 Processor -- start 2024-04-18T10:06:03.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 -- start start 2024-04-18T10:06:03.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdadc0b8db0 0x7fdadc13d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdadc13ebd0 0x7fdadc09fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdadc147f60 0x7fdadc1419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdadc150460 con 0x7fdadc0b8db0 2024-04-18T10:06:03.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fdadc1502e0 con 0x7fdadc147f60 2024-04-18T10:06:03.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae3efe640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fdadc1505e0 con 0x7fdadc13ebd0 2024-04-18T10:06:03.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae1c9a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdadc0b8db0 0x7fdadc13d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae1c9a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdadc0b8db0 0x7fdadc13d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53104/0 (socket says 172.21.15.62:53104) 2024-04-18T10:06:03.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae1c9a640 1 -- 172.21.15.62:0/890994198 learned_addr learned my addr 172.21.15.62:0/890994198 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:03.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae1499640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdadc13ebd0 0x7fdadc09fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae249b640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdadc147f60 0x7fdadc1419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae1c9a640 1 -- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdadc13ebd0 msgr2=0x7fdadc09fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae1c9a640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdadc13ebd0 0x7fdadc09fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae1c9a640 1 -- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdadc147f60 msgr2=0x7fdadc1419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae1c9a640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdadc147f60 0x7fdadc1419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae1c9a640 1 -- 172.21.15.62:0/890994198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdadc13dc00 con 0x7fdadc0b8db0 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdae1c9a640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdadc0b8db0 0x7fdadc13d6c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdad800c7f0 tx=0x7fdad800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.490+0000 7fdad27fc640 1 -- 172.21.15.62:0/890994198 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad8015070 con 0x7fdadc0b8db0 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.494+0000 7fdad27fc640 1 -- 172.21.15.62:0/890994198 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdad8004240 con 0x7fdadc0b8db0 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.494+0000 7fdae3efe640 1 -- 172.21.15.62:0/890994198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdadc0a0430 con 0x7fdadc0b8db0 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.494+0000 7fdad27fc640 1 -- 172.21.15.62:0/890994198 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad8003620 con 0x7fdadc0b8db0 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.494+0000 7fdae3efe640 1 -- 172.21.15.62:0/890994198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdadc1deac0 con 0x7fdadc0b8db0 2024-04-18T10:06:03.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.494+0000 7fdae3efe640 1 -- 172.21.15.62:0/890994198 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdadc069230 con 0x7fdadc0b8db0 2024-04-18T10:06:03.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.494+0000 7fdad27fc640 1 -- 172.21.15.62:0/890994198 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdad80039d0 con 0x7fdadc0b8db0 2024-04-18T10:06:03.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.494+0000 7fdad27fc640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdac0068380 0x7fdac006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.494+0000 7fdad27fc640 1 -- 172.21.15.62:0/890994198 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdad8086990 con 0x7fdadc0b8db0 2024-04-18T10:06:03.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.494+0000 7fdae1499640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdac0068380 0x7fdac006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.494+0000 7fdae1499640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdac0068380 0x7fdac006a840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0ad000 tx=0x7fdacc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:03.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.498+0000 7fdad27fc640 1 -- 172.21.15.62:0/890994198 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdad8056fe0 con 0x7fdadc0b8db0 2024-04-18T10:06:03.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.658+0000 7fdae3efe640 1 -- 172.21.15.62:0/890994198 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdadc1ded00 con 0x7fdac0068380 2024-04-18T10:06:03.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.674+0000 7fdad27fc640 1 -- 172.21.15.62:0/890994198 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251955 (secure 0 0 0) 0x7fdadc1ded00 con 0x7fdac0068380 2024-04-18T10:06:03.676 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:03.677 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-18T10:06:03.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.678+0000 7fdae3efe640 1 -- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdac0068380 msgr2=0x7fdac006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.678+0000 7fdae3efe640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdac0068380 0x7fdac006a840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdacc0ad000 tx=0x7fdacc008040 comp rx=0 tx=0).stop 2024-04-18T10:06:03.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.678+0000 7fdae3efe640 1 -- 172.21.15.62:0/890994198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdadc0b8db0 msgr2=0x7fdadc13d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.678+0000 7fdae3efe640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdadc0b8db0 0x7fdadc13d6c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdad800c7f0 tx=0x7fdad800ccb0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.678+0000 7fdae3efe640 1 -- 172.21.15.62:0/890994198 shutdown_connections 2024-04-18T10:06:03.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.678+0000 7fdae3efe640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdac0068380 0x7fdac006a840 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.678+0000 7fdae3efe640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdadc147f60 0x7fdadc1419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.678+0000 7fdae3efe640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdadc13ebd0 0x7fdadc09fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.678+0000 7fdae3efe640 1 --2- 172.21.15.62:0/890994198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdadc0b8db0 0x7fdadc13d6c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.678+0000 7fdae3efe640 1 -- 172.21.15.62:0/890994198 >> 172.21.15.62:0/890994198 conn(0x7fdadc09c250 msgr2=0x7fdadc074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:03.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.678+0000 7fdae3efe640 1 -- 172.21.15.62:0/890994198 shutdown_connections 2024-04-18T10:06:03.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.678+0000 7fdae3efe640 1 -- 172.21.15.62:0/890994198 wait complete. 2024-04-18T10:06:03.694 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-04-18T10:06:03.126165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219116,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755604,"statfs":{"total":773094113280,"available":772869738496,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.931952"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242380+0000","last_change":"2024-04-18T10:05:59.270038+0000","last_active":"2024-04-18T10:06:00.242380+0000","last_peered":"2024-04-18T10:06:00.242380+0000","last_clean":"2024-04-18T10:06:00.242380+0000","last_became_active":"2024-04-18T10:05:59.269226+0000","last_became_peered":"2024-04-18T10:05:59.269226+0000","last_unstale":"2024-04-18T10:06:00.242380+0000","last_undegraded":"2024-04-18T10:06:00.242380+0000","last_fullsized":"2024-04-18T10:06:00.242380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:45:43.670227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232794+0000","last_change":"2024-04-18T10:05:59.230274+0000","last_active":"2024-04-18T10:06:00.232794+0000","last_peered":"2024-04-18T10:06:00.232794+0000","last_clean":"2024-04-18T10:06:00.232794+0000","last_became_active":"2024-04-18T10:05:59.229843+0000","last_became_peered":"2024-04-18T10:05:59.229843+0000","last_unstale":"2024-04-18T10:06:00.232794+0000","last_undegraded":"2024-04-18T10:06:00.232794+0000","last_fullsized":"2024-04-18T10:06:00.232794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384628+0000","last_change":"2024-04-18T10:05:59.228858+0000","last_active":"2024-04-18T10:06:00.384628+0000","last_peered":"2024-04-18T10:06:00.384628+0000","last_clean":"2024-04-18T10:06:00.384628+0000","last_became_active":"2024-04-18T10:05:59.228382+0000","last_became_peered":"2024-04-18T10:05:59.228382+0000","last_unstale":"2024-04-18T10:06:00.384628+0000","last_undegraded":"2024-04-18T10:06:00.384628+0000","last_fullsized":"2024-04-18T10:06:00.384628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240458+0000","last_change":"2024-04-18T10:05:59.240142+0000","last_active":"2024-04-18T10:06:00.240458+0000","last_peered":"2024-04-18T10:06:00.240458+0000","last_clean":"2024-04-18T10:06:00.240458+0000","last_became_active":"2024-04-18T10:05:59.239298+0000","last_became_peered":"2024-04-18T10:05:59.239298+0000","last_unstale":"2024-04-18T10:06:00.240458+0000","last_undegraded":"2024-04-18T10:06:00.240458+0000","last_fullsized":"2024-04-18T10:06:00.240458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:49.354229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239594+0000","last_change":"2024-04-18T10:05:59.235434+0000","last_active":"2024-04-18T10:06:00.239594+0000","last_peered":"2024-04-18T10:06:00.239594+0000","last_clean":"2024-04-18T10:06:00.239594+0000","last_became_active":"2024-04-18T10:05:59.234575+0000","last_became_peered":"2024-04-18T10:05:59.234575+0000","last_unstale":"2024-04-18T10:06:00.239594+0000","last_undegraded":"2024-04-18T10:06:00.239594+0000","last_fullsized":"2024-04-18T10:06:00.239594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:34:28.871228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238452+0000","last_change":"2024-04-18T10:05:59.232083+0000","last_active":"2024-04-18T10:06:00.238452+0000","last_peered":"2024-04-18T10:06:00.238452+0000","last_clean":"2024-04-18T10:06:00.238452+0000","last_became_active":"2024-04-18T10:05:59.231313+0000","last_became_peered":"2024-04-18T10:05:59.231313+0000","last_unstale":"2024-04-18T10:06:00.238452+0000","last_undegraded":"2024-04-18T10:06:00.238452+0000","last_fullsized":"2024-04-18T10:06:00.238452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238876+0000","last_change":"2024-04-18T10:05:59.250962+0000","last_active":"2024-04-18T10:06:00.238876+0000","last_peered":"2024-04-18T10:06:00.238876+0000","last_clean":"2024-04-18T10:06:00.238876+0000","last_became_active":"2024-04-18T10:05:59.250059+0000","last_became_peered":"2024-04-18T10:05:59.250059+0000","last_unstale":"2024-04-18T10:06:00.238876+0000","last_undegraded":"2024-04-18T10:06:00.238876+0000","last_fullsized":"2024-04-18T10:06:00.238876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:12:17.167339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383437+0000","last_change":"2024-04-18T10:05:59.256210+0000","last_active":"2024-04-18T10:06:00.383437+0000","last_peered":"2024-04-18T10:06:00.383437+0000","last_clean":"2024-04-18T10:06:00.383437+0000","last_became_active":"2024-04-18T10:05:59.255975+0000","last_became_peered":"2024-04-18T10:05:59.255975+0000","last_unstale":"2024-04-18T10:06:00.383437+0000","last_undegraded":"2024-04-18T10:06:00.383437+0000","last_fullsized":"2024-04-18T10:06:00.383437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:49:34.764977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242471+0000","last_change":"2024-04-18T10:05:59.272589+0000","last_active":"2024-04-18T10:06:00.242471+0000","last_peered":"2024-04-18T10:06:00.242471+0000","last_clean":"2024-04-18T10:06:00.242471+0000","last_became_active":"2024-04-18T10:05:59.271883+0000","last_became_peered":"2024-04-18T10:05:59.271883+0000","last_unstale":"2024-04-18T10:06:00.242471+0000","last_undegraded":"2024-04-18T10:06:00.242471+0000","last_fullsized":"2024-04-18T10:06:00.242471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:24:15.929261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243230+0000","last_change":"2024-04-18T10:05:59.261146+0000","last_active":"2024-04-18T10:06:00.243230+0000","last_peered":"2024-04-18T10:06:00.243230+0000","last_clean":"2024-04-18T10:06:00.243230+0000","last_became_active":"2024-04-18T10:05:59.260455+0000","last_became_peered":"2024-04-18T10:05:59.260455+0000","last_unstale":"2024-04-18T10:06:00.243230+0000","last_undegraded":"2024-04-18T10:06:00.243230+0000","last_fullsized":"2024-04-18T10:06:00.243230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.244095+0000","last_change":"2024-04-18T10:05:59.260958+0000","last_active":"2024-04-18T10:06:00.244095+0000","last_peered":"2024-04-18T10:06:00.244095+0000","last_clean":"2024-04-18T10:06:00.244095+0000","last_became_active":"2024-04-18T10:05:59.260251+0000","last_became_peered":"2024-04-18T10:05:59.260251+0000","last_unstale":"2024-04-18T10:06:00.244095+0000","last_undegraded":"2024-04-18T10:06:00.244095+0000","last_fullsized":"2024-04-18T10:06:00.244095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384238+0000","last_change":"2024-04-18T10:05:59.238407+0000","last_active":"2024-04-18T10:06:00.384238+0000","last_peered":"2024-04-18T10:06:00.384238+0000","last_clean":"2024-04-18T10:06:00.384238+0000","last_became_active":"2024-04-18T10:05:59.236530+0000","last_became_peered":"2024-04-18T10:05:59.236530+0000","last_unstale":"2024-04-18T10:06:00.384238+0000","last_undegraded":"2024-04-18T10:06:00.384238+0000","last_fullsized":"2024-04-18T10:06:00.384238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:51.419833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384926+0000","last_change":"2024-04-18T10:05:59.243032+0000","last_active":"2024-04-18T10:06:00.384926+0000","last_peered":"2024-04-18T10:06:00.384926+0000","last_clean":"2024-04-18T10:06:00.384926+0000","last_became_active":"2024-04-18T10:05:59.242716+0000","last_became_peered":"2024-04-18T10:05:59.242716+0000","last_unstale":"2024-04-18T10:06:00.384926+0000","last_undegraded":"2024-04-18T10:06:00.384926+0000","last_fullsized":"2024-04-18T10:06:00.384926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384052+0000","last_change":"2024-04-18T10:05:59.240754+0000","last_active":"2024-04-18T10:06:00.384052+0000","last_peered":"2024-04-18T10:06:00.384052+0000","last_clean":"2024-04-18T10:06:00.384052+0000","last_became_active":"2024-04-18T10:05:59.239992+0000","last_became_peered":"2024-04-18T10:05:59.239992+0000","last_unstale":"2024-04-18T10:06:00.384052+0000","last_undegraded":"2024-04-18T10:06:00.384052+0000","last_fullsized":"2024-04-18T10:06:00.384052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232972+0000","last_change":"2024-04-18T10:05:59.224563+0000","last_active":"2024-04-18T10:06:00.232972+0000","last_peered":"2024-04-18T10:06:00.232972+0000","last_clean":"2024-04-18T10:06:00.232972+0000","last_became_active":"2024-04-18T10:05:59.224115+0000","last_became_peered":"2024-04-18T10:05:59.224115+0000","last_unstale":"2024-04-18T10:06:00.232972+0000","last_undegraded":"2024-04-18T10:06:00.232972+0000","last_fullsized":"2024-04-18T10:06:00.232972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238461+0000","last_change":"2024-04-18T10:05:59.232777+0000","last_active":"2024-04-18T10:06:00.238461+0000","last_peered":"2024-04-18T10:06:00.238461+0000","last_clean":"2024-04-18T10:06:00.238461+0000","last_became_active":"2024-04-18T10:05:59.231188+0000","last_became_peered":"2024-04-18T10:05:59.231188+0000","last_unstale":"2024-04-18T10:06:00.238461+0000","last_undegraded":"2024-04-18T10:06:00.238461+0000","last_fullsized":"2024-04-18T10:06:00.238461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243840+0000","last_change":"2024-04-18T10:05:59.264529+0000","last_active":"2024-04-18T10:06:00.243840+0000","last_peered":"2024-04-18T10:06:00.243840+0000","last_clean":"2024-04-18T10:06:00.243840+0000","last_became_active":"2024-04-18T10:05:59.260972+0000","last_became_peered":"2024-04-18T10:05:59.260972+0000","last_unstale":"2024-04-18T10:06:00.243840+0000","last_undegraded":"2024-04-18T10:06:00.243840+0000","last_fullsized":"2024-04-18T10:06:00.243840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:24:15.929261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385847+0000","last_change":"2024-04-18T10:05:59.262102+0000","last_active":"2024-04-18T10:06:00.385847+0000","last_peered":"2024-04-18T10:06:00.385847+0000","last_clean":"2024-04-18T10:06:00.385847+0000","last_became_active":"2024-04-18T10:05:59.261796+0000","last_became_peered":"2024-04-18T10:05:59.261796+0000","last_unstale":"2024-04-18T10:06:00.385847+0000","last_undegraded":"2024-04-18T10:06:00.385847+0000","last_fullsized":"2024-04-18T10:06:00.385847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:51.419833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241173+0000","last_change":"2024-04-18T10:05:59.256981+0000","last_active":"2024-04-18T10:06:00.241173+0000","last_peered":"2024-04-18T10:06:00.241173+0000","last_clean":"2024-04-18T10:06:00.241173+0000","last_became_active":"2024-04-18T10:05:59.252309+0000","last_became_peered":"2024-04-18T10:05:59.252309+0000","last_unstale":"2024-04-18T10:06:00.241173+0000","last_undegraded":"2024-04-18T10:06:00.241173+0000","last_fullsized":"2024-04-18T10:06:00.241173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:16:23.709955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243343+0000","last_change":"2024-04-18T10:05:59.266020+0000","last_active":"2024-04-18T10:06:00.243343+0000","last_peered":"2024-04-18T10:06:00.243343+0000","last_clean":"2024-04-18T10:06:00.243343+0000","last_became_active":"2024-04-18T10:05:59.265361+0000","last_became_peered":"2024-04-18T10:05:59.265361+0000","last_unstale":"2024-04-18T10:06:00.243343+0000","last_undegraded":"2024-04-18T10:06:00.243343+0000","last_fullsized":"2024-04-18T10:06:00.243343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384730+0000","last_change":"2024-04-18T10:05:59.262190+0000","last_active":"2024-04-18T10:06:00.384730+0000","last_peered":"2024-04-18T10:06:00.384730+0000","last_clean":"2024-04-18T10:06:00.384730+0000","last_became_active":"2024-04-18T10:05:59.261866+0000","last_became_peered":"2024-04-18T10:05:59.261866+0000","last_unstale":"2024-04-18T10:06:00.384730+0000","last_undegraded":"2024-04-18T10:06:00.384730+0000","last_fullsized":"2024-04-18T10:06:00.384730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:49:34.764977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238819+0000","last_change":"2024-04-18T10:05:59.241285+0000","last_active":"2024-04-18T10:06:00.238819+0000","last_peered":"2024-04-18T10:06:00.238819+0000","last_clean":"2024-04-18T10:06:00.238819+0000","last_became_active":"2024-04-18T10:05:59.239568+0000","last_became_peered":"2024-04-18T10:05:59.239568+0000","last_unstale":"2024-04-18T10:06:00.238819+0000","last_undegraded":"2024-04-18T10:06:00.238819+0000","last_fullsized":"2024-04-18T10:06:00.238819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:12:17.167339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385177+0000","last_change":"2024-04-18T10:05:59.264180+0000","last_active":"2024-04-18T10:06:00.385177+0000","last_peered":"2024-04-18T10:06:00.385177+0000","last_clean":"2024-04-18T10:06:00.385177+0000","last_became_active":"2024-04-18T10:05:59.263866+0000","last_became_peered":"2024-04-18T10:05:59.263866+0000","last_unstale":"2024-04-18T10:06:00.385177+0000","last_undegraded":"2024-04-18T10:06:00.385177+0000","last_fullsized":"2024-04-18T10:06:00.385177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238829+0000","last_change":"2024-04-18T10:05:59.246355+0000","last_active":"2024-04-18T10:06:00.238829+0000","last_peered":"2024-04-18T10:06:00.238829+0000","last_clean":"2024-04-18T10:06:00.238829+0000","last_became_active":"2024-04-18T10:05:59.239576+0000","last_became_peered":"2024-04-18T10:05:59.239576+0000","last_unstale":"2024-04-18T10:06:00.238829+0000","last_undegraded":"2024-04-18T10:06:00.238829+0000","last_fullsized":"2024-04-18T10:06:00.238829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:27:56.094004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386366+0000","last_change":"2024-04-18T10:05:59.264586+0000","last_active":"2024-04-18T10:06:00.386366+0000","last_peered":"2024-04-18T10:06:00.386366+0000","last_clean":"2024-04-18T10:06:00.386366+0000","last_became_active":"2024-04-18T10:05:59.264215+0000","last_became_peered":"2024-04-18T10:05:59.264215+0000","last_unstale":"2024-04-18T10:06:00.386366+0000","last_undegraded":"2024-04-18T10:06:00.386366+0000","last_fullsized":"2024-04-18T10:06:00.386366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:27:56.094004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242998+0000","last_change":"2024-04-18T10:05:59.243781+0000","last_active":"2024-04-18T10:06:00.242998+0000","last_peered":"2024-04-18T10:06:00.242998+0000","last_clean":"2024-04-18T10:06:00.242998+0000","last_became_active":"2024-04-18T10:05:59.243092+0000","last_became_peered":"2024-04-18T10:05:59.243092+0000","last_unstale":"2024-04-18T10:06:00.242998+0000","last_undegraded":"2024-04-18T10:06:00.242998+0000","last_fullsized":"2024-04-18T10:06:00.242998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:49:34.764977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232294+0000","last_change":"2024-04-18T10:05:59.234128+0000","last_active":"2024-04-18T10:06:00.232294+0000","last_peered":"2024-04-18T10:06:00.232294+0000","last_clean":"2024-04-18T10:06:00.232294+0000","last_became_active":"2024-04-18T10:05:59.233742+0000","last_became_peered":"2024-04-18T10:05:59.233742+0000","last_unstale":"2024-04-18T10:06:00.232294+0000","last_undegraded":"2024-04-18T10:06:00.232294+0000","last_fullsized":"2024-04-18T10:06:00.232294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386224+0000","last_change":"2024-04-18T10:05:59.264826+0000","last_active":"2024-04-18T10:06:00.386224+0000","last_peered":"2024-04-18T10:06:00.386224+0000","last_clean":"2024-04-18T10:06:00.386224+0000","last_became_active":"2024-04-18T10:05:59.263655+0000","last_became_peered":"2024-04-18T10:05:59.263655+0000","last_unstale":"2024-04-18T10:06:00.386224+0000","last_undegraded":"2024-04-18T10:06:00.386224+0000","last_fullsized":"2024-04-18T10:06:00.386224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:12:17.167339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383983+0000","last_change":"2024-04-18T10:05:59.235032+0000","last_active":"2024-04-18T10:06:00.383983+0000","last_peered":"2024-04-18T10:06:00.383983+0000","last_clean":"2024-04-18T10:06:00.383983+0000","last_became_active":"2024-04-18T10:05:59.234660+0000","last_became_peered":"2024-04-18T10:05:59.234660+0000","last_unstale":"2024-04-18T10:06:00.383983+0000","last_undegraded":"2024-04-18T10:06:00.383983+0000","last_fullsized":"2024-04-18T10:06:00.383983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239182+0000","last_change":"2024-04-18T10:05:59.241915+0000","last_active":"2024-04-18T10:06:00.239182+0000","last_peered":"2024-04-18T10:06:00.239182+0000","last_clean":"2024-04-18T10:06:00.239182+0000","last_became_active":"2024-04-18T10:05:59.240519+0000","last_became_peered":"2024-04-18T10:05:59.240519+0000","last_unstale":"2024-04-18T10:06:00.239182+0000","last_undegraded":"2024-04-18T10:06:00.239182+0000","last_fullsized":"2024-04-18T10:06:00.239182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:34:28.871228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385003+0000","last_change":"2024-04-18T10:05:59.246146+0000","last_active":"2024-04-18T10:06:00.385003+0000","last_peered":"2024-04-18T10:06:00.385003+0000","last_clean":"2024-04-18T10:06:00.385003+0000","last_became_active":"2024-04-18T10:05:59.245800+0000","last_became_peered":"2024-04-18T10:05:59.245800+0000","last_unstale":"2024-04-18T10:06:00.385003+0000","last_undegraded":"2024-04-18T10:06:00.385003+0000","last_fullsized":"2024-04-18T10:06:00.385003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:22.118304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242883+0000","last_change":"2024-04-18T10:05:59.239684+0000","last_active":"2024-04-18T10:06:00.242883+0000","last_peered":"2024-04-18T10:06:00.242883+0000","last_clean":"2024-04-18T10:06:00.242883+0000","last_became_active":"2024-04-18T10:05:59.233494+0000","last_became_peered":"2024-04-18T10:05:59.233494+0000","last_unstale":"2024-04-18T10:06:00.242883+0000","last_undegraded":"2024-04-18T10:06:00.242883+0000","last_fullsized":"2024-04-18T10:06:00.242883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:23:54.569394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383451+0000","last_change":"2024-04-18T10:05:59.219492+0000","last_active":"2024-04-18T10:06:00.383451+0000","last_peered":"2024-04-18T10:06:00.383451+0000","last_clean":"2024-04-18T10:06:00.383451+0000","last_became_active":"2024-04-18T10:05:59.218737+0000","last_became_peered":"2024-04-18T10:05:59.218737+0000","last_unstale":"2024-04-18T10:06:00.383451+0000","last_undegraded":"2024-04-18T10:06:00.383451+0000","last_fullsized":"2024-04-18T10:06:00.383451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:33.010147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386355+0000","last_change":"2024-04-18T10:05:58.253056+0000","last_active":"2024-04-18T10:06:00.386355+0000","last_peered":"2024-04-18T10:06:00.386355+0000","last_clean":"2024-04-18T10:06:00.386355+0000","last_became_active":"2024-04-18T10:05:58.252737+0000","last_became_peered":"2024-04-18T10:05:58.252737+0000","last_unstale":"2024-04-18T10:06:00.386355+0000","last_undegraded":"2024-04-18T10:06:00.386355+0000","last_fullsized":"2024-04-18T10:06:00.386355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384430+0000","last_change":"2024-04-18T10:05:59.263750+0000","last_active":"2024-04-18T10:06:00.384430+0000","last_peered":"2024-04-18T10:06:00.384430+0000","last_clean":"2024-04-18T10:06:00.384430+0000","last_became_active":"2024-04-18T10:05:59.263538+0000","last_became_peered":"2024-04-18T10:05:59.263538+0000","last_unstale":"2024-04-18T10:06:00.384430+0000","last_undegraded":"2024-04-18T10:06:00.384430+0000","last_fullsized":"2024-04-18T10:06:00.384430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243880+0000","last_change":"2024-04-18T10:05:58.243026+0000","last_active":"2024-04-18T10:06:00.243880+0000","last_peered":"2024-04-18T10:06:00.243880+0000","last_clean":"2024-04-18T10:06:00.243880+0000","last_became_active":"2024-04-18T10:05:58.240506+0000","last_became_peered":"2024-04-18T10:05:58.240506+0000","last_unstale":"2024-04-18T10:06:00.243880+0000","last_undegraded":"2024-04-18T10:06:00.243880+0000","last_fullsized":"2024-04-18T10:06:00.243880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385279+0000","last_change":"2024-04-18T10:05:58.243128+0000","last_active":"2024-04-18T10:06:00.385279+0000","last_peered":"2024-04-18T10:06:00.385279+0000","last_clean":"2024-04-18T10:06:00.385279+0000","last_became_active":"2024-04-18T10:05:58.242694+0000","last_became_peered":"2024-04-18T10:05:58.242694+0000","last_unstale":"2024-04-18T10:06:00.385279+0000","last_undegraded":"2024-04-18T10:06:00.385279+0000","last_fullsized":"2024-04-18T10:06:00.385279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:52:26.587547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383890+0000","last_change":"2024-04-18T10:05:59.232704+0000","last_active":"2024-04-18T10:06:00.383890+0000","last_peered":"2024-04-18T10:06:00.383890+0000","last_clean":"2024-04-18T10:06:00.383890+0000","last_became_active":"2024-04-18T10:05:59.231762+0000","last_became_peered":"2024-04-18T10:05:59.231762+0000","last_unstale":"2024-04-18T10:06:00.383890+0000","last_undegraded":"2024-04-18T10:06:00.383890+0000","last_fullsized":"2024-04-18T10:06:00.383890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239055+0000","last_change":"2024-04-18T10:05:58.231745+0000","last_active":"2024-04-18T10:06:00.239055+0000","last_peered":"2024-04-18T10:06:00.239055+0000","last_clean":"2024-04-18T10:06:00.239055+0000","last_became_active":"2024-04-18T10:05:58.230994+0000","last_became_peered":"2024-04-18T10:05:58.230994+0000","last_unstale":"2024-04-18T10:06:00.239055+0000","last_undegraded":"2024-04-18T10:06:00.239055+0000","last_fullsized":"2024-04-18T10:06:00.239055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232217+0000","last_change":"2024-04-18T10:05:59.232550+0000","last_active":"2024-04-18T10:06:00.232217+0000","last_peered":"2024-04-18T10:06:00.232217+0000","last_clean":"2024-04-18T10:06:00.232217+0000","last_became_active":"2024-04-18T10:05:59.231955+0000","last_became_peered":"2024-04-18T10:05:59.231955+0000","last_unstale":"2024-04-18T10:06:00.232217+0000","last_undegraded":"2024-04-18T10:06:00.232217+0000","last_fullsized":"2024-04-18T10:06:00.232217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243516+0000","last_change":"2024-04-18T10:05:58.244903+0000","last_active":"2024-04-18T10:06:00.243516+0000","last_peered":"2024-04-18T10:06:00.243516+0000","last_clean":"2024-04-18T10:06:00.243516+0000","last_became_active":"2024-04-18T10:05:58.244287+0000","last_became_peered":"2024-04-18T10:05:58.244287+0000","last_unstale":"2024-04-18T10:06:00.243516+0000","last_undegraded":"2024-04-18T10:06:00.243516+0000","last_fullsized":"2024-04-18T10:06:00.243516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240811+0000","last_change":"2024-04-18T10:05:59.238308+0000","last_active":"2024-04-18T10:06:00.240811+0000","last_peered":"2024-04-18T10:06:00.240811+0000","last_clean":"2024-04-18T10:06:00.240811+0000","last_became_active":"2024-04-18T10:05:59.237422+0000","last_became_peered":"2024-04-18T10:05:59.237422+0000","last_unstale":"2024-04-18T10:06:00.240811+0000","last_undegraded":"2024-04-18T10:06:00.240811+0000","last_fullsized":"2024-04-18T10:06:00.240811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:28:59.797065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240802+0000","last_change":"2024-04-18T10:05:58.245925+0000","last_active":"2024-04-18T10:06:00.240802+0000","last_peered":"2024-04-18T10:06:00.240802+0000","last_clean":"2024-04-18T10:06:00.240802+0000","last_became_active":"2024-04-18T10:05:58.244713+0000","last_became_peered":"2024-04-18T10:05:58.244713+0000","last_unstale":"2024-04-18T10:06:00.240802+0000","last_undegraded":"2024-04-18T10:06:00.240802+0000","last_fullsized":"2024-04-18T10:06:00.240802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384230+0000","last_change":"2024-04-18T10:05:59.252961+0000","last_active":"2024-04-18T10:06:00.384230+0000","last_peered":"2024-04-18T10:06:00.384230+0000","last_clean":"2024-04-18T10:06:00.384230+0000","last_became_active":"2024-04-18T10:05:59.252672+0000","last_became_peered":"2024-04-18T10:05:59.252672+0000","last_unstale":"2024-04-18T10:06:00.384230+0000","last_undegraded":"2024-04-18T10:06:00.384230+0000","last_fullsized":"2024-04-18T10:06:00.384230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232406+0000","last_change":"2024-04-18T10:05:58.235997+0000","last_active":"2024-04-18T10:06:00.232406+0000","last_peered":"2024-04-18T10:06:00.232406+0000","last_clean":"2024-04-18T10:06:00.232406+0000","last_became_active":"2024-04-18T10:05:58.235330+0000","last_became_peered":"2024-04-18T10:05:58.235330+0000","last_unstale":"2024-04-18T10:06:00.232406+0000","last_undegraded":"2024-04-18T10:06:00.232406+0000","last_fullsized":"2024-04-18T10:06:00.232406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:21.107270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384433+0000","last_change":"2024-04-18T10:05:59.257176+0000","last_active":"2024-04-18T10:06:00.384433+0000","last_peered":"2024-04-18T10:06:00.384433+0000","last_clean":"2024-04-18T10:06:00.384433+0000","last_became_active":"2024-04-18T10:05:59.256452+0000","last_became_peered":"2024-04-18T10:05:59.256452+0000","last_unstale":"2024-04-18T10:06:00.384433+0000","last_undegraded":"2024-04-18T10:06:00.384433+0000","last_fullsized":"2024-04-18T10:06:00.384433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:04:27.662085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243463+0000","last_change":"2024-04-18T10:05:58.239207+0000","last_active":"2024-04-18T10:06:00.243463+0000","last_peered":"2024-04-18T10:06:00.243463+0000","last_clean":"2024-04-18T10:06:00.243463+0000","last_became_active":"2024-04-18T10:05:58.238349+0000","last_became_peered":"2024-04-18T10:05:58.238349+0000","last_unstale":"2024-04-18T10:06:00.243463+0000","last_undegraded":"2024-04-18T10:06:00.243463+0000","last_fullsized":"2024-04-18T10:06:00.243463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239372+0000","last_change":"2024-04-18T10:05:59.254087+0000","last_active":"2024-04-18T10:06:00.239372+0000","last_peered":"2024-04-18T10:06:00.239372+0000","last_clean":"2024-04-18T10:06:00.239372+0000","last_became_active":"2024-04-18T10:05:59.253296+0000","last_became_peered":"2024-04-18T10:05:59.253296+0000","last_unstale":"2024-04-18T10:06:00.239372+0000","last_undegraded":"2024-04-18T10:06:00.239372+0000","last_fullsized":"2024-04-18T10:06:00.239372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:27:56.094004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384701+0000","last_change":"2024-04-18T10:05:58.236375+0000","last_active":"2024-04-18T10:06:00.384701+0000","last_peered":"2024-04-18T10:06:00.384701+0000","last_clean":"2024-04-18T10:06:00.384701+0000","last_became_active":"2024-04-18T10:05:58.236066+0000","last_became_peered":"2024-04-18T10:05:58.236066+0000","last_unstale":"2024-04-18T10:06:00.384701+0000","last_undegraded":"2024-04-18T10:06:00.384701+0000","last_fullsized":"2024-04-18T10:06:00.384701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:52:26.587547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383621+0000","last_change":"2024-04-18T10:05:59.262667+0000","last_active":"2024-04-18T10:06:00.383621+0000","last_peered":"2024-04-18T10:06:00.383621+0000","last_clean":"2024-04-18T10:06:00.383621+0000","last_became_active":"2024-04-18T10:05:59.262365+0000","last_became_peered":"2024-04-18T10:05:59.262365+0000","last_unstale":"2024-04-18T10:06:00.383621+0000","last_undegraded":"2024-04-18T10:06:00.383621+0000","last_fullsized":"2024-04-18T10:06:00.383621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:23:54.569394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239431+0000","last_change":"2024-04-18T10:05:58.243114+0000","last_active":"2024-04-18T10:06:00.239431+0000","last_peered":"2024-04-18T10:06:00.239431+0000","last_clean":"2024-04-18T10:06:00.239431+0000","last_became_active":"2024-04-18T10:05:58.242102+0000","last_became_peered":"2024-04-18T10:05:58.242102+0000","last_unstale":"2024-04-18T10:06:00.239431+0000","last_undegraded":"2024-04-18T10:06:00.239431+0000","last_fullsized":"2024-04-18T10:06:00.239431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239878+0000","last_change":"2024-04-18T10:05:59.274245+0000","last_active":"2024-04-18T10:06:00.239878+0000","last_peered":"2024-04-18T10:06:00.239878+0000","last_clean":"2024-04-18T10:06:00.239878+0000","last_became_active":"2024-04-18T10:05:59.273707+0000","last_became_peered":"2024-04-18T10:05:59.273707+0000","last_unstale":"2024-04-18T10:06:00.239878+0000","last_undegraded":"2024-04-18T10:06:00.239878+0000","last_fullsized":"2024-04-18T10:06:00.239878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:49.354229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243436+0000","last_change":"2024-04-18T10:05:58.242951+0000","last_active":"2024-04-18T10:06:00.243436+0000","last_peered":"2024-04-18T10:06:00.243436+0000","last_clean":"2024-04-18T10:06:00.243436+0000","last_became_active":"2024-04-18T10:05:58.241438+0000","last_became_peered":"2024-04-18T10:05:58.241438+0000","last_unstale":"2024-04-18T10:06:00.243436+0000","last_undegraded":"2024-04-18T10:06:00.243436+0000","last_fullsized":"2024-04-18T10:06:00.243436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243154+0000","last_change":"2024-04-18T10:05:59.233588+0000","last_active":"2024-04-18T10:06:00.243154+0000","last_peered":"2024-04-18T10:06:00.243154+0000","last_clean":"2024-04-18T10:06:00.243154+0000","last_became_active":"2024-04-18T10:05:59.232831+0000","last_became_peered":"2024-04-18T10:05:59.232831+0000","last_unstale":"2024-04-18T10:06:00.243154+0000","last_undegraded":"2024-04-18T10:06:00.243154+0000","last_fullsized":"2024-04-18T10:06:00.243154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384620+0000","last_change":"2024-04-18T10:05:58.223796+0000","last_active":"2024-04-18T10:06:00.384620+0000","last_peered":"2024-04-18T10:06:00.384620+0000","last_clean":"2024-04-18T10:06:00.384620+0000","last_became_active":"2024-04-18T10:05:58.223248+0000","last_became_peered":"2024-04-18T10:05:58.223248+0000","last_unstale":"2024-04-18T10:06:00.384620+0000","last_undegraded":"2024-04-18T10:06:00.384620+0000","last_fullsized":"2024-04-18T10:06:00.384620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:52:26.587547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240028+0000","last_change":"2024-04-18T10:05:59.277176+0000","last_active":"2024-04-18T10:06:00.240028+0000","last_peered":"2024-04-18T10:06:00.240028+0000","last_clean":"2024-04-18T10:06:00.240028+0000","last_became_active":"2024-04-18T10:05:59.276386+0000","last_became_peered":"2024-04-18T10:05:59.276386+0000","last_unstale":"2024-04-18T10:06:00.240028+0000","last_undegraded":"2024-04-18T10:06:00.240028+0000","last_fullsized":"2024-04-18T10:06:00.240028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:28:59.797065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384638+0000","last_change":"2024-04-18T10:05:58.233338+0000","last_active":"2024-04-18T10:06:00.384638+0000","last_peered":"2024-04-18T10:06:00.384638+0000","last_clean":"2024-04-18T10:06:00.384638+0000","last_became_active":"2024-04-18T10:05:58.231906+0000","last_became_peered":"2024-04-18T10:05:58.231906+0000","last_unstale":"2024-04-18T10:06:00.384638+0000","last_undegraded":"2024-04-18T10:06:00.384638+0000","last_fullsized":"2024-04-18T10:06:00.384638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:06:57.988393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241935+0000","last_change":"2024-04-18T10:05:59.275977+0000","last_active":"2024-04-18T10:06:00.241935+0000","last_peered":"2024-04-18T10:06:00.241935+0000","last_clean":"2024-04-18T10:06:00.241935+0000","last_became_active":"2024-04-18T10:05:59.275386+0000","last_became_peered":"2024-04-18T10:05:59.275386+0000","last_unstale":"2024-04-18T10:06:00.241935+0000","last_undegraded":"2024-04-18T10:06:00.241935+0000","last_fullsized":"2024-04-18T10:06:00.241935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:58:16.367066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386132+0000","last_change":"2024-04-18T10:05:58.229897+0000","last_active":"2024-04-18T10:06:00.386132+0000","last_peered":"2024-04-18T10:06:00.386132+0000","last_clean":"2024-04-18T10:06:00.386132+0000","last_became_active":"2024-04-18T10:05:58.229451+0000","last_became_peered":"2024-04-18T10:05:58.229451+0000","last_unstale":"2024-04-18T10:06:00.386132+0000","last_undegraded":"2024-04-18T10:06:00.386132+0000","last_fullsized":"2024-04-18T10:06:00.386132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242763+0000","last_change":"2024-04-18T10:05:59.276815+0000","last_active":"2024-04-18T10:06:00.242763+0000","last_peered":"2024-04-18T10:06:00.242763+0000","last_clean":"2024-04-18T10:06:00.242763+0000","last_became_active":"2024-04-18T10:05:59.276272+0000","last_became_peered":"2024-04-18T10:05:59.276272+0000","last_unstale":"2024-04-18T10:06:00.242763+0000","last_undegraded":"2024-04-18T10:06:00.242763+0000","last_fullsized":"2024-04-18T10:06:00.242763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:44:45.775310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232479+0000","last_change":"2024-04-18T10:05:58.230942+0000","last_active":"2024-04-18T10:06:00.232479+0000","last_peered":"2024-04-18T10:06:00.232479+0000","last_clean":"2024-04-18T10:06:00.232479+0000","last_became_active":"2024-04-18T10:05:58.230373+0000","last_became_peered":"2024-04-18T10:05:58.230373+0000","last_unstale":"2024-04-18T10:06:00.232479+0000","last_undegraded":"2024-04-18T10:06:00.232479+0000","last_fullsized":"2024-04-18T10:06:00.232479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:50.408798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241359+0000","last_change":"2024-04-18T10:05:59.234199+0000","last_active":"2024-04-18T10:06:00.241359+0000","last_peered":"2024-04-18T10:06:00.241359+0000","last_clean":"2024-04-18T10:06:00.241359+0000","last_became_active":"2024-04-18T10:05:59.233412+0000","last_became_peered":"2024-04-18T10:05:59.233412+0000","last_unstale":"2024-04-18T10:06:00.241359+0000","last_undegraded":"2024-04-18T10:06:00.241359+0000","last_fullsized":"2024-04-18T10:06:00.241359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:16:23.709955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385242+0000","last_change":"2024-04-18T10:05:58.222111+0000","last_active":"2024-04-18T10:06:00.385242+0000","last_peered":"2024-04-18T10:06:00.385242+0000","last_clean":"2024-04-18T10:06:00.385242+0000","last_became_active":"2024-04-18T10:05:58.221783+0000","last_became_peered":"2024-04-18T10:05:58.221783+0000","last_unstale":"2024-04-18T10:06:00.385242+0000","last_undegraded":"2024-04-18T10:06:00.385242+0000","last_fullsized":"2024-04-18T10:06:00.385242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240501+0000","last_change":"2024-04-18T10:05:59.261884+0000","last_active":"2024-04-18T10:06:00.240501+0000","last_peered":"2024-04-18T10:06:00.240501+0000","last_clean":"2024-04-18T10:06:00.240501+0000","last_became_active":"2024-04-18T10:05:59.261147+0000","last_became_peered":"2024-04-18T10:05:59.261147+0000","last_unstale":"2024-04-18T10:06:00.240501+0000","last_undegraded":"2024-04-18T10:06:00.240501+0000","last_fullsized":"2024-04-18T10:06:00.240501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:44:42.861177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238296+0000","last_change":"2024-04-18T10:05:49.183850+0000","last_active":"2024-04-18T10:06:00.238296+0000","last_peered":"2024-04-18T10:06:00.238296+0000","last_clean":"2024-04-18T10:06:00.238296+0000","last_became_active":"2024-04-18T10:05:49.181471+0000","last_became_peered":"2024-04-18T10:05:49.181471+0000","last_unstale":"2024-04-18T10:06:00.238296+0000","last_undegraded":"2024-04-18T10:06:00.238296+0000","last_fullsized":"2024-04-18T10:06:00.238296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_clean_scrub_stamp":"2024-04-18T10:05:47.511486+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:06:51.214547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239978+0000","last_change":"2024-04-18T10:05:58.228257+0000","last_active":"2024-04-18T10:06:00.239978+0000","last_peered":"2024-04-18T10:06:00.239978+0000","last_clean":"2024-04-18T10:06:00.239978+0000","last_became_active":"2024-04-18T10:05:58.224594+0000","last_became_peered":"2024-04-18T10:05:58.224594+0000","last_unstale":"2024-04-18T10:06:00.239978+0000","last_undegraded":"2024-04-18T10:06:00.239978+0000","last_fullsized":"2024-04-18T10:06:00.239978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385522+0000","last_change":"2024-04-18T10:05:59.256229+0000","last_active":"2024-04-18T10:06:00.385522+0000","last_peered":"2024-04-18T10:06:00.385522+0000","last_clean":"2024-04-18T10:06:00.385522+0000","last_became_active":"2024-04-18T10:05:59.255894+0000","last_became_peered":"2024-04-18T10:05:59.255894+0000","last_unstale":"2024-04-18T10:06:00.385522+0000","last_undegraded":"2024-04-18T10:06:00.385522+0000","last_fullsized":"2024-04-18T10:06:00.385522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238224+0000","last_change":"2024-04-18T10:05:58.254200+0000","last_active":"2024-04-18T10:06:00.238224+0000","last_peered":"2024-04-18T10:06:00.238224+0000","last_clean":"2024-04-18T10:06:00.238224+0000","last_became_active":"2024-04-18T10:05:58.253365+0000","last_became_peered":"2024-04-18T10:05:58.253365+0000","last_unstale":"2024-04-18T10:06:00.238224+0000","last_undegraded":"2024-04-18T10:06:00.238224+0000","last_fullsized":"2024-04-18T10:06:00.238224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383217+0000","last_change":"2024-04-18T10:05:59.264229+0000","last_active":"2024-04-18T10:06:00.383217+0000","last_peered":"2024-04-18T10:06:00.383217+0000","last_clean":"2024-04-18T10:06:00.383217+0000","last_became_active":"2024-04-18T10:05:59.263577+0000","last_became_peered":"2024-04-18T10:05:59.263577+0000","last_unstale":"2024-04-18T10:06:00.383217+0000","last_undegraded":"2024-04-18T10:06:00.383217+0000","last_fullsized":"2024-04-18T10:06:00.383217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:37:20.126666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385257+0000","last_change":"2024-04-18T10:05:58.235466+0000","last_active":"2024-04-18T10:06:00.385257+0000","last_peered":"2024-04-18T10:06:00.385257+0000","last_clean":"2024-04-18T10:06:00.385257+0000","last_became_active":"2024-04-18T10:05:58.234823+0000","last_became_peered":"2024-04-18T10:05:58.234823+0000","last_unstale":"2024-04-18T10:06:00.385257+0000","last_undegraded":"2024-04-18T10:06:00.385257+0000","last_fullsized":"2024-04-18T10:06:00.385257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232451+0000","last_change":"2024-04-18T10:05:59.230409+0000","last_active":"2024-04-18T10:06:00.232451+0000","last_peered":"2024-04-18T10:06:00.232451+0000","last_clean":"2024-04-18T10:06:00.232451+0000","last_became_active":"2024-04-18T10:05:59.228899+0000","last_became_peered":"2024-04-18T10:05:59.228899+0000","last_unstale":"2024-04-18T10:06:00.232451+0000","last_undegraded":"2024-04-18T10:06:00.232451+0000","last_fullsized":"2024-04-18T10:06:00.232451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239789+0000","last_change":"2024-04-18T10:05:58.232028+0000","last_active":"2024-04-18T10:06:00.239789+0000","last_peered":"2024-04-18T10:06:00.239789+0000","last_clean":"2024-04-18T10:06:00.239789+0000","last_became_active":"2024-04-18T10:05:58.229311+0000","last_became_peered":"2024-04-18T10:05:58.229311+0000","last_unstale":"2024-04-18T10:06:00.239789+0000","last_undegraded":"2024-04-18T10:06:00.239789+0000","last_fullsized":"2024-04-18T10:06:00.239789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383872+0000","last_change":"2024-04-18T10:05:59.248629+0000","last_active":"2024-04-18T10:06:00.383872+0000","last_peered":"2024-04-18T10:06:00.383872+0000","last_clean":"2024-04-18T10:06:00.383872+0000","last_became_active":"2024-04-18T10:05:59.247619+0000","last_became_peered":"2024-04-18T10:05:59.247619+0000","last_unstale":"2024-04-18T10:06:00.383872+0000","last_undegraded":"2024-04-18T10:06:00.383872+0000","last_fullsized":"2024-04-18T10:06:00.383872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:35.965832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232588+0000","last_change":"2024-04-18T10:05:58.224570+0000","last_active":"2024-04-18T10:06:00.232588+0000","last_peered":"2024-04-18T10:06:00.232588+0000","last_clean":"2024-04-18T10:06:00.232588+0000","last_became_active":"2024-04-18T10:05:58.224216+0000","last_became_peered":"2024-04-18T10:05:58.224216+0000","last_unstale":"2024-04-18T10:06:00.232588+0000","last_undegraded":"2024-04-18T10:06:00.232588+0000","last_fullsized":"2024-04-18T10:06:00.232588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242688+0000","last_change":"2024-04-18T10:05:59.246784+0000","last_active":"2024-04-18T10:06:00.242688+0000","last_peered":"2024-04-18T10:06:00.242688+0000","last_clean":"2024-04-18T10:06:00.242688+0000","last_became_active":"2024-04-18T10:05:59.243518+0000","last_became_peered":"2024-04-18T10:05:59.243518+0000","last_unstale":"2024-04-18T10:06:00.242688+0000","last_undegraded":"2024-04-18T10:06:00.242688+0000","last_fullsized":"2024-04-18T10:06:00.242688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:45:43.670227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241271+0000","last_change":"2024-04-18T10:05:58.258428+0000","last_active":"2024-04-18T10:06:00.241271+0000","last_peered":"2024-04-18T10:06:00.241271+0000","last_clean":"2024-04-18T10:06:00.241271+0000","last_became_active":"2024-04-18T10:05:58.257648+0000","last_became_peered":"2024-04-18T10:05:58.257648+0000","last_unstale":"2024-04-18T10:06:00.241271+0000","last_undegraded":"2024-04-18T10:06:00.241271+0000","last_fullsized":"2024-04-18T10:06:00.241271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:15:05.853314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232543+0000","last_change":"2024-04-18T10:05:59.229368+0000","last_active":"2024-04-18T10:06:00.232543+0000","last_peered":"2024-04-18T10:06:00.232543+0000","last_clean":"2024-04-18T10:06:00.232543+0000","last_became_active":"2024-04-18T10:05:59.228912+0000","last_became_peered":"2024-04-18T10:05:59.228912+0000","last_unstale":"2024-04-18T10:06:00.232543+0000","last_undegraded":"2024-04-18T10:06:00.232543+0000","last_fullsized":"2024-04-18T10:06:00.232543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241350+0000","last_change":"2024-04-18T10:05:58.243743+0000","last_active":"2024-04-18T10:06:00.241350+0000","last_peered":"2024-04-18T10:06:00.241350+0000","last_clean":"2024-04-18T10:06:00.241350+0000","last_became_active":"2024-04-18T10:05:58.243448+0000","last_became_peered":"2024-04-18T10:05:58.243448+0000","last_unstale":"2024-04-18T10:06:00.241350+0000","last_undegraded":"2024-04-18T10:06:00.241350+0000","last_fullsized":"2024-04-18T10:06:00.241350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:04:26.651051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386763+0000","last_change":"2024-04-18T10:05:59.262383+0000","last_active":"2024-04-18T10:06:00.386763+0000","last_peered":"2024-04-18T10:06:00.386763+0000","last_clean":"2024-04-18T10:06:00.386763+0000","last_became_active":"2024-04-18T10:05:59.262039+0000","last_became_peered":"2024-04-18T10:05:59.262039+0000","last_unstale":"2024-04-18T10:06:00.386763+0000","last_undegraded":"2024-04-18T10:06:00.386763+0000","last_fullsized":"2024-04-18T10:06:00.386763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:49.354229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243760+0000","last_change":"2024-04-18T10:05:58.251779+0000","last_active":"2024-04-18T10:06:00.243760+0000","last_peered":"2024-04-18T10:06:00.243760+0000","last_clean":"2024-04-18T10:06:00.243760+0000","last_became_active":"2024-04-18T10:05:58.251062+0000","last_became_peered":"2024-04-18T10:05:58.251062+0000","last_unstale":"2024-04-18T10:06:00.243760+0000","last_undegraded":"2024-04-18T10:06:00.243760+0000","last_fullsized":"2024-04-18T10:06:00.243760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384029+0000","last_change":"2024-04-18T10:05:59.240684+0000","last_active":"2024-04-18T10:06:00.384029+0000","last_peered":"2024-04-18T10:06:00.384029+0000","last_clean":"2024-04-18T10:06:00.384029+0000","last_became_active":"2024-04-18T10:05:59.240231+0000","last_became_peered":"2024-04-18T10:05:59.240231+0000","last_unstale":"2024-04-18T10:06:00.384029+0000","last_undegraded":"2024-04-18T10:06:00.384029+0000","last_fullsized":"2024-04-18T10:06:00.384029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238119+0000","last_change":"2024-04-18T10:05:58.254872+0000","last_active":"2024-04-18T10:06:00.238119+0000","last_peered":"2024-04-18T10:06:00.238119+0000","last_clean":"2024-04-18T10:06:00.238119+0000","last_became_active":"2024-04-18T10:05:58.252439+0000","last_became_peered":"2024-04-18T10:05:58.252439+0000","last_unstale":"2024-04-18T10:06:00.238119+0000","last_undegraded":"2024-04-18T10:06:00.238119+0000","last_fullsized":"2024-04-18T10:06:00.238119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384090+0000","last_change":"2024-04-18T10:05:59.254583+0000","last_active":"2024-04-18T10:06:00.384090+0000","last_peered":"2024-04-18T10:06:00.384090+0000","last_clean":"2024-04-18T10:06:00.384090+0000","last_became_active":"2024-04-18T10:05:59.253836+0000","last_became_peered":"2024-04-18T10:05:59.253836+0000","last_unstale":"2024-04-18T10:06:00.384090+0000","last_undegraded":"2024-04-18T10:06:00.384090+0000","last_fullsized":"2024-04-18T10:06:00.384090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:22.118304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232357+0000","last_change":"2024-04-18T10:05:58.248261+0000","last_active":"2024-04-18T10:06:00.232357+0000","last_peered":"2024-04-18T10:06:00.232357+0000","last_clean":"2024-04-18T10:06:00.232357+0000","last_became_active":"2024-04-18T10:05:58.247708+0000","last_became_peered":"2024-04-18T10:05:58.247708+0000","last_unstale":"2024-04-18T10:06:00.232357+0000","last_undegraded":"2024-04-18T10:06:00.232357+0000","last_fullsized":"2024-04-18T10:06:00.232357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384524+0000","last_change":"2024-04-18T10:05:59.238345+0000","last_active":"2024-04-18T10:06:00.384524+0000","last_peered":"2024-04-18T10:06:00.384524+0000","last_clean":"2024-04-18T10:06:00.384524+0000","last_became_active":"2024-04-18T10:05:59.237905+0000","last_became_peered":"2024-04-18T10:05:59.237905+0000","last_unstale":"2024-04-18T10:06:00.384524+0000","last_undegraded":"2024-04-18T10:06:00.384524+0000","last_fullsized":"2024-04-18T10:06:00.384524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232345+0000","last_change":"2024-04-18T10:05:58.229628+0000","last_active":"2024-04-18T10:06:00.232345+0000","last_peered":"2024-04-18T10:06:00.232345+0000","last_clean":"2024-04-18T10:06:00.232345+0000","last_became_active":"2024-04-18T10:05:58.229323+0000","last_became_peered":"2024-04-18T10:05:58.229323+0000","last_unstale":"2024-04-18T10:06:00.232345+0000","last_undegraded":"2024-04-18T10:06:00.232345+0000","last_fullsized":"2024-04-18T10:06:00.232345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385760+0000","last_change":"2024-04-18T10:05:59.232981+0000","last_active":"2024-04-18T10:06:00.385760+0000","last_peered":"2024-04-18T10:06:00.385760+0000","last_clean":"2024-04-18T10:06:00.385760+0000","last_became_active":"2024-04-18T10:05:59.232662+0000","last_became_peered":"2024-04-18T10:05:59.232662+0000","last_unstale":"2024-04-18T10:06:00.385760+0000","last_undegraded":"2024-04-18T10:06:00.385760+0000","last_fullsized":"2024-04-18T10:06:00.385760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239926+0000","last_change":"2024-04-18T10:05:58.258738+0000","last_active":"2024-04-18T10:06:00.239926+0000","last_peered":"2024-04-18T10:06:00.239926+0000","last_clean":"2024-04-18T10:06:00.239926+0000","last_became_active":"2024-04-18T10:05:58.257527+0000","last_became_peered":"2024-04-18T10:05:58.257527+0000","last_unstale":"2024-04-18T10:06:00.239926+0000","last_undegraded":"2024-04-18T10:06:00.239926+0000","last_fullsized":"2024-04-18T10:06:00.239926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:21.107270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243622+0000","last_change":"2024-04-18T10:05:59.264420+0000","last_active":"2024-04-18T10:06:00.243622+0000","last_peered":"2024-04-18T10:06:00.243622+0000","last_clean":"2024-04-18T10:06:00.243622+0000","last_became_active":"2024-04-18T10:05:59.263738+0000","last_became_peered":"2024-04-18T10:05:59.263738+0000","last_unstale":"2024-04-18T10:06:00.243622+0000","last_undegraded":"2024-04-18T10:06:00.243622+0000","last_fullsized":"2024-04-18T10:06:00.243622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242313+0000","last_change":"2024-04-18T10:05:59.222327+0000","last_active":"2024-04-18T10:06:00.242313+0000","last_peered":"2024-04-18T10:06:00.242313+0000","last_clean":"2024-04-18T10:06:00.242313+0000","last_became_active":"2024-04-18T10:05:59.221623+0000","last_became_peered":"2024-04-18T10:05:59.221623+0000","last_unstale":"2024-04-18T10:06:00.242313+0000","last_undegraded":"2024-04-18T10:06:00.242313+0000","last_fullsized":"2024-04-18T10:06:00.242313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:58:16.367066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239652+0000","last_change":"2024-04-18T10:05:58.255714+0000","last_active":"2024-04-18T10:06:00.239652+0000","last_peered":"2024-04-18T10:06:00.239652+0000","last_clean":"2024-04-18T10:06:00.239652+0000","last_became_active":"2024-04-18T10:05:58.255115+0000","last_became_peered":"2024-04-18T10:05:58.255115+0000","last_unstale":"2024-04-18T10:06:00.239652+0000","last_undegraded":"2024-04-18T10:06:00.239652+0000","last_fullsized":"2024-04-18T10:06:00.239652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:50.408798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240843+0000","last_change":"2024-04-18T10:05:59.244685+0000","last_active":"2024-04-18T10:06:00.240843+0000","last_peered":"2024-04-18T10:06:00.240843+0000","last_clean":"2024-04-18T10:06:00.240843+0000","last_became_active":"2024-04-18T10:05:59.243811+0000","last_became_peered":"2024-04-18T10:05:59.243811+0000","last_unstale":"2024-04-18T10:06:00.240843+0000","last_undegraded":"2024-04-18T10:06:00.240843+0000","last_fullsized":"2024-04-18T10:06:00.240843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:44:42.861177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241014+0000","last_change":"2024-04-18T10:05:58.256922+0000","last_active":"2024-04-18T10:06:00.241014+0000","last_peered":"2024-04-18T10:06:00.241014+0000","last_clean":"2024-04-18T10:06:00.241014+0000","last_became_active":"2024-04-18T10:05:58.255255+0000","last_became_peered":"2024-04-18T10:05:58.255255+0000","last_unstale":"2024-04-18T10:06:00.241014+0000","last_undegraded":"2024-04-18T10:06:00.241014+0000","last_fullsized":"2024-04-18T10:06:00.241014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:04:26.651051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383632+0000","last_change":"2024-04-18T10:05:59.253025+0000","last_active":"2024-04-18T10:06:00.383632+0000","last_peered":"2024-04-18T10:06:00.383632+0000","last_clean":"2024-04-18T10:06:00.383632+0000","last_became_active":"2024-04-18T10:05:59.252708+0000","last_became_peered":"2024-04-18T10:05:59.252708+0000","last_unstale":"2024-04-18T10:06:00.383632+0000","last_undegraded":"2024-04-18T10:06:00.383632+0000","last_fullsized":"2024-04-18T10:06:00.383632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:17:23.117921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386955+0000","last_change":"2024-04-18T10:05:58.234122+0000","last_active":"2024-04-18T10:06:00.386955+0000","last_peered":"2024-04-18T10:06:00.386955+0000","last_clean":"2024-04-18T10:06:00.386955+0000","last_became_active":"2024-04-18T10:05:58.233320+0000","last_became_peered":"2024-04-18T10:05:58.233320+0000","last_unstale":"2024-04-18T10:06:00.386955+0000","last_undegraded":"2024-04-18T10:06:00.386955+0000","last_fullsized":"2024-04-18T10:06:00.386955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386581+0000","last_change":"2024-04-18T10:05:59.244230+0000","last_active":"2024-04-18T10:06:00.386581+0000","last_peered":"2024-04-18T10:06:00.386581+0000","last_clean":"2024-04-18T10:06:00.386581+0000","last_became_active":"2024-04-18T10:05:59.243879+0000","last_became_peered":"2024-04-18T10:05:59.243879+0000","last_unstale":"2024-04-18T10:06:00.386581+0000","last_undegraded":"2024-04-18T10:06:00.386581+0000","last_fullsized":"2024-04-18T10:06:00.386581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:34:28.871228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383948+0000","last_change":"2024-04-18T10:05:58.247996+0000","last_active":"2024-04-18T10:06:00.383948+0000","last_peered":"2024-04-18T10:06:00.383948+0000","last_clean":"2024-04-18T10:06:00.383948+0000","last_became_active":"2024-04-18T10:05:58.247612+0000","last_became_peered":"2024-04-18T10:05:58.247612+0000","last_unstale":"2024-04-18T10:06:00.383948+0000","last_undegraded":"2024-04-18T10:06:00.383948+0000","last_fullsized":"2024-04-18T10:06:00.383948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:13:49.398609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T10:06:03.698 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-18T10:06:03.698 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-18T10:06:03.698 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-18T10:06:03.699 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-18T10:06:03.699 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-18T10:06:03.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.866+0000 7f86afc30640 1 Processor -- start 2024-04-18T10:06:03.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86afc30640 1 -- start start 2024-04-18T10:06:03.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86afc30640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f86a8067a70 0x7f86a809edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86afc30640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f86a80b46f0 0x7f86a81450f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86afc30640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a8067e50 con 0x7f86a80b6210 2024-04-18T10:06:03.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86afc30640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f86a80b4b60 con 0x7f86a8067a70 2024-04-18T10:06:03.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86afc30640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f86a80af220 con 0x7f86a80b46f0 2024-04-18T10:06:03.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86ad9cc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f86a8067a70 0x7f86a809edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86ad9cc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f86a8067a70 0x7f86a809edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53106/0 (socket says 172.21.15.62:53106) 2024-04-18T10:06:03.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86ad9cc640 1 -- 172.21.15.62:0/2723820271 learned_addr learned my addr 172.21.15.62:0/2723820271 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:03.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86ae1cd640 1 --2- 172.21.15.62:0/2723820271 >> v2:172.21.15.145:3300/0 conn(0x7f86a80b46f0 0x7f86a81450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86ad9cc640 1 -- 172.21.15.62:0/2723820271 >> v2:172.21.15.145:3300/0 conn(0x7f86a80b46f0 msgr2=0x7f86a81450f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:03.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86ad9cc640 1 --2- 172.21.15.62:0/2723820271 >> v2:172.21.15.145:3300/0 conn(0x7f86a80b46f0 0x7f86a81450f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86ad9cc640 1 -- 172.21.15.62:0/2723820271 >> v1:172.21.15.145:6789/0 conn(0x7f86a80b6210 legacy=0x7f86a80af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:03.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86ad9cc640 1 -- 172.21.15.62:0/2723820271 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a80aef40 con 0x7f86a8067a70 2024-04-18T10:06:03.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86ae1cd640 1 --2- 172.21.15.62:0/2723820271 >> v2:172.21.15.145:3300/0 conn(0x7f86a80b46f0 0x7f86a81450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.870+0000 7f86ad9cc640 1 --2- 172.21.15.62:0/2723820271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86a8067a70 0x7f86a809edc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f86a80b0570 tx=0x7f86a401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=118c1a66c402aa6d server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ac9ca640 1 -- 172.21.15.62:0/2723820271 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a40b0070 con 0x7f86a8067a70 2024-04-18T10:06:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ac9ca640 1 -- 172.21.15.62:0/2723820271 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86a40a4e50 con 0x7f86a8067a70 2024-04-18T10:06:03.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ac9ca640 1 -- 172.21.15.62:0/2723820271 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a40ab820 con 0x7f86a8067a70 2024-04-18T10:06:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 -- 172.21.15.62:0/2723820271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86a8067a70 msgr2=0x7f86a809edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 --2- 172.21.15.62:0/2723820271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86a8067a70 0x7f86a809edc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f86a80b0570 tx=0x7f86a401c3c0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 -- 172.21.15.62:0/2723820271 shutdown_connections 2024-04-18T10:06:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 --2- 172.21.15.62:0/2723820271 >> v2:172.21.15.145:3300/0 conn(0x7f86a80b46f0 0x7f86a81450f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 --2- 172.21.15.62:0/2723820271 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86a8067a70 0x7f86a809edc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 -- 172.21.15.62:0/2723820271 >> 172.21.15.62:0/2723820271 conn(0x7f86a806c380 msgr2=0x7f86a80b9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 -- 172.21.15.62:0/2723820271 shutdown_connections 2024-04-18T10:06:03.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 -- 172.21.15.62:0/2723820271 wait complete. 2024-04-18T10:06:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 Processor -- start 2024-04-18T10:06:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 -- start start 2024-04-18T10:06:03.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86a8067a70 0x7f86a8072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86a80b46f0 0x7f86a80b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86a81e0410 0x7f86a81e08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f86a8151e70 con 0x7f86a81e0410 2024-04-18T10:06:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f86a8151cf0 con 0x7f86a80b46f0 2024-04-18T10:06:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f86a8151ff0 con 0x7f86a8067a70 2024-04-18T10:06:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ae1cd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86a81e0410 0x7f86a81e08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ae1cd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86a81e0410 0x7f86a81e08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53112/0 (socket says 172.21.15.62:53112) 2024-04-18T10:06:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ae1cd640 1 -- 172.21.15.62:0/2584500344 learned_addr learned my addr 172.21.15.62:0/2584500344 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ad1cb640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86a80b46f0 0x7f86a80b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ad9cc640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86a8067a70 0x7f86a8072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ae1cd640 1 -- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86a8067a70 msgr2=0x7f86a8072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ae1cd640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86a8067a70 0x7f86a8072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ae1cd640 1 -- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86a80b46f0 msgr2=0x7f86a80b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ae1cd640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86a80b46f0 0x7f86a80b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ae1cd640 1 -- 172.21.15.62:0/2584500344 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a814efd0 con 0x7f86a81e0410 2024-04-18T10:06:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86ae1cd640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86a81e0410 0x7f86a81e08c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f86a000cf20 tx=0x7f86a000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f869e7fc640 1 -- 172.21.15.62:0/2584500344 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a001a070 con 0x7f86a81e0410 2024-04-18T10:06:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f869e7fc640 1 -- 172.21.15.62:0/2584500344 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86a000d840 con 0x7f86a81e0410 2024-04-18T10:06:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 -- 172.21.15.62:0/2584500344 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a81dff10 con 0x7f86a81e0410 2024-04-18T10:06:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f869e7fc640 1 -- 172.21.15.62:0/2584500344 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a000dc30 con 0x7f86a81e0410 2024-04-18T10:06:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.874+0000 7f86afc30640 1 -- 172.21.15.62:0/2584500344 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a80b2330 con 0x7f86a81e0410 2024-04-18T10:06:03.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.878+0000 7f86afc30640 1 -- 172.21.15.62:0/2584500344 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a8068b90 con 0x7f86a81e0410 2024-04-18T10:06:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.878+0000 7f869e7fc640 1 -- 172.21.15.62:0/2584500344 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f86a001d070 con 0x7f86a81e0410 2024-04-18T10:06:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.878+0000 7f869e7fc640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f868c068380 0x7f868c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:03.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.878+0000 7f869e7fc640 1 -- 172.21.15.62:0/2584500344 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f86a00871e0 con 0x7f86a81e0410 2024-04-18T10:06:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.878+0000 7f86ad9cc640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f868c068380 0x7f868c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:03.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.878+0000 7f86ad9cc640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f868c068380 0x7f868c06a840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f86a40ad000 tx=0x7f86a40a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:03.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:03.882+0000 7f869e7fc640 1 -- 172.21.15.62:0/2584500344 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f86a0057830 con 0x7f86a81e0410 2024-04-18T10:06:04.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.086+0000 7f86afc30640 1 -- 172.21.15.62:0/2584500344 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f86a80b2610 con 0x7f86a81e0410 2024-04-18T10:06:04.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.090+0000 7f869e7fc640 1 -- 172.21.15.62:0/2584500344 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+168090 (secure 0 0 0) 0x7f86a0057130 con 0x7f86a81e0410 2024-04-18T10:06:04.091 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:04.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.094+0000 7f86afc30640 1 -- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f868c068380 msgr2=0x7f868c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.094+0000 7f86afc30640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f868c068380 0x7f868c06a840 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f86a40ad000 tx=0x7f86a40a6000 comp rx=0 tx=0).stop 2024-04-18T10:06:04.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.094+0000 7f86afc30640 1 -- 172.21.15.62:0/2584500344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86a81e0410 msgr2=0x7f86a81e08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.094+0000 7f86afc30640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86a81e0410 0x7f86a81e08c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f86a000cf20 tx=0x7f86a000a7d0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.094+0000 7f86afc30640 1 -- 172.21.15.62:0/2584500344 shutdown_connections 2024-04-18T10:06:04.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.094+0000 7f86afc30640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f868c068380 0x7f868c06a840 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.094+0000 7f86afc30640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f86a81e0410 0x7f86a81e08c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.094+0000 7f86afc30640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86a80b46f0 0x7f86a80b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.094+0000 7f86afc30640 1 --2- 172.21.15.62:0/2584500344 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86a8067a70 0x7f86a8072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.094+0000 7f86afc30640 1 -- 172.21.15.62:0/2584500344 >> 172.21.15.62:0/2584500344 conn(0x7f86a806c380 msgr2=0x7f86a80b5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:04.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.094+0000 7f86afc30640 1 -- 172.21.15.62:0/2584500344 shutdown_connections 2024-04-18T10:06:04.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.094+0000 7f86afc30640 1 -- 172.21.15.62:0/2584500344 wait complete. 2024-04-18T10:06:04.110 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":4004518230},{"type":"v1","addr":"172.21.15.145:6833","nonce":4004518230}]},"active_addr":"172.21.15.145:6833/4004518230","active_change":"2024-04-18T10:05:39.109627+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":106986912}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":1864560233}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":414478862}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":4157086979}]}]} 2024-04-18T10:06:04.113 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-18T10:06:04.114 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-18T10:06:04.114 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:04.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.238+0000 7f46d1140640 1 Processor -- start 2024-04-18T10:06:04.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.238+0000 7f46d1140640 1 -- start start 2024-04-18T10:06:04.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f46cc0b65a0 0x7f46cc141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f46cc142110 0x7f46cc14b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46cc0b6980 con 0x7f46cc14b7e0 2024-04-18T10:06:04.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f46cc0b4790 con 0x7f46cc0b65a0 2024-04-18T10:06:04.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f46cc147e50 con 0x7f46cc142110 2024-04-18T10:06:04.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46cb59e640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f46cc14b7e0 0x7f46cc1482e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54884/0 (socket says 172.21.15.62:54884) 2024-04-18T10:06:04.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46cb59e640 1 -- 172.21.15.62:0/3771134373 learned_addr learned my addr 172.21.15.62:0/3771134373 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:04.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46cad9d640 1 --2- 172.21.15.62:0/3771134373 >> v2:172.21.15.62:3300/0 conn(0x7f46cc0b65a0 0x7f46cc141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46ca59c640 1 --2- 172.21.15.62:0/3771134373 >> v2:172.21.15.145:3300/0 conn(0x7f46cc142110 0x7f46cc14b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46c9d9b640 1 -- 172.21.15.62:0/3771134373 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 573802520 0 0) 0x7f46cc0b6980 con 0x7f46cc14b7e0 2024-04-18T10:06:04.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46c9d9b640 1 -- 172.21.15.62:0/3771134373 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46b4003530 con 0x7f46cc14b7e0 2024-04-18T10:06:04.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46cad9d640 1 -- 172.21.15.62:0/3771134373 >> v2:172.21.15.145:3300/0 conn(0x7f46cc142110 msgr2=0x7f46cc14b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:04.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46cad9d640 1 --2- 172.21.15.62:0/3771134373 >> v2:172.21.15.145:3300/0 conn(0x7f46cc142110 0x7f46cc14b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46cad9d640 1 -- 172.21.15.62:0/3771134373 >> v1:172.21.15.62:6789/0 conn(0x7f46cc14b7e0 legacy=0x7f46cc1482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46cad9d640 1 -- 172.21.15.62:0/3771134373 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc14c180 con 0x7f46cc0b65a0 2024-04-18T10:06:04.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46cad9d640 1 --2- 172.21.15.62:0/3771134373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46cc0b65a0 0x7f46cc141bd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0be200 tx=0x7f46bc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e119101f64823efd server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46c9d9b640 1 -- 172.21.15.62:0/3771134373 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc0b0070 con 0x7f46cc0b65a0 2024-04-18T10:06:04.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46c9d9b640 1 -- 172.21.15.62:0/3771134373 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46bc0ab600 con 0x7f46cc0b65a0 2024-04-18T10:06:04.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46c9d9b640 1 -- 172.21.15.62:0/3771134373 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc0ab9d0 con 0x7f46cc0b65a0 2024-04-18T10:06:04.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 -- 172.21.15.62:0/3771134373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46cc0b65a0 msgr2=0x7f46cc141bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 --2- 172.21.15.62:0/3771134373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46cc0b65a0 0x7f46cc141bd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0be200 tx=0x7f46bc01c3f0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 -- 172.21.15.62:0/3771134373 shutdown_connections 2024-04-18T10:06:04.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 --2- 172.21.15.62:0/3771134373 >> v2:172.21.15.145:3300/0 conn(0x7f46cc142110 0x7f46cc14b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 --2- 172.21.15.62:0/3771134373 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46cc0b65a0 0x7f46cc141bd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 -- 172.21.15.62:0/3771134373 >> 172.21.15.62:0/3771134373 conn(0x7f46cc0680c0 msgr2=0x7f46cc0bd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:04.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 -- 172.21.15.62:0/3771134373 shutdown_connections 2024-04-18T10:06:04.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.242+0000 7f46d1140640 1 -- 172.21.15.62:0/3771134373 wait complete. 2024-04-18T10:06:04.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46d1140640 1 Processor -- start 2024-04-18T10:06:04.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46d1140640 1 -- start start 2024-04-18T10:06:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46d1140640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46cc0b65a0 0x7f46cc145010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46d1140640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46cc142110 0x7f46cc13f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46d1140640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46cc14b7e0 0x7f46cc13f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46d1140640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc1551e0 con 0x7f46cc0b65a0 2024-04-18T10:06:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46d1140640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f46cc155060 con 0x7f46cc142110 2024-04-18T10:06:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46d1140640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f46cc155360 con 0x7f46cc14b7e0 2024-04-18T10:06:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46cad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46cc0b65a0 0x7f46cc145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46cad9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46cc0b65a0 0x7f46cc145010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53130/0 (socket says 172.21.15.62:53130) 2024-04-18T10:06:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46cad9d640 1 -- 172.21.15.62:0/1764747000 learned_addr learned my addr 172.21.15.62:0/1764747000 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46ca59c640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46cc142110 0x7f46cc13f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46cb59e640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46cc14b7e0 0x7f46cc13f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46cad9d640 1 -- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46cc14b7e0 msgr2=0x7f46cc13f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46cad9d640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46cc14b7e0 0x7f46cc13f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46cad9d640 1 -- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46cc142110 msgr2=0x7f46cc13f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46cad9d640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46cc142110 0x7f46cc13f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46cad9d640 1 -- 172.21.15.62:0/1764747000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46cc140bf0 con 0x7f46cc0b65a0 2024-04-18T10:06:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46cb59e640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46cc14b7e0 0x7f46cc13f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46cad9d640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46cc0b65a0 0x7f46cc145010 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0ae040 tx=0x7f46bc01ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46b37fe640 1 -- 172.21.15.62:0/1764747000 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc0b0050 con 0x7f46cc0b65a0 2024-04-18T10:06:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46b37fe640 1 -- 172.21.15.62:0/1764747000 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46bc0bc070 con 0x7f46cc0b65a0 2024-04-18T10:06:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46b37fe640 1 -- 172.21.15.62:0/1764747000 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46bc0b5a30 con 0x7f46cc0b65a0 2024-04-18T10:06:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46d1140640 1 -- 172.21.15.62:0/1764747000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46cc140ed0 con 0x7f46cc0b65a0 2024-04-18T10:06:04.247 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46d1140640 1 -- 172.21.15.62:0/1764747000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46cc141640 con 0x7f46cc0b65a0 2024-04-18T10:06:04.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.246+0000 7f46d1140640 1 -- 172.21.15.62:0/1764747000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46cc068b90 con 0x7f46cc0b65a0 2024-04-18T10:06:04.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.250+0000 7f46b37fe640 1 -- 172.21.15.62:0/1764747000 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f46bc0b5bd0 con 0x7f46cc0b65a0 2024-04-18T10:06:04.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.250+0000 7f46b37fe640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f46b8068380 0x7f46b806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.250+0000 7f46b37fe640 1 -- 172.21.15.62:0/1764747000 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f46bc121250 con 0x7f46cc0b65a0 2024-04-18T10:06:04.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.250+0000 7f46ca59c640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f46b8068380 0x7f46b806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.248 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.250+0000 7f46ca59c640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f46b8068380 0x7f46b806a840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f46cc152600 tx=0x7f46c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.254+0000 7f46b37fe640 1 -- 172.21.15.62:0/1764747000 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f46bc016070 con 0x7f46cc0b65a0 2024-04-18T10:06:04.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.418+0000 7f46d1140640 1 -- 172.21.15.62:0/1764747000 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46cc1456d0 con 0x7f46cc0b65a0 2024-04-18T10:06:04.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46b37fe640 1 -- 172.21.15.62:0/1764747000 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f46bc0f18a0 con 0x7f46cc0b65a0 2024-04-18T10:06:04.420 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:04.420 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:04.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46d1140640 1 -- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f46b8068380 msgr2=0x7f46b806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46d1140640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f46b8068380 0x7f46b806a840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f46cc152600 tx=0x7f46c0008040 comp rx=0 tx=0).stop 2024-04-18T10:06:04.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46d1140640 1 -- 172.21.15.62:0/1764747000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46cc0b65a0 msgr2=0x7f46cc145010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46d1140640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46cc0b65a0 0x7f46cc145010 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0ae040 tx=0x7f46bc01ac30 comp rx=0 tx=0).stop 2024-04-18T10:06:04.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46d1140640 1 -- 172.21.15.62:0/1764747000 shutdown_connections 2024-04-18T10:06:04.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46d1140640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f46b8068380 0x7f46b806a840 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46d1140640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46cc14b7e0 0x7f46cc13f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46d1140640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46cc142110 0x7f46cc13f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46d1140640 1 --2- 172.21.15.62:0/1764747000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f46cc0b65a0 0x7f46cc145010 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46d1140640 1 -- 172.21.15.62:0/1764747000 >> 172.21.15.62:0/1764747000 conn(0x7f46cc0680c0 msgr2=0x7f46cc0b4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:04.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46d1140640 1 -- 172.21.15.62:0/1764747000 shutdown_connections 2024-04-18T10:06:04.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.422+0000 7f46d1140640 1 -- 172.21.15.62:0/1764747000 wait complete. 2024-04-18T10:06:04.439 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-18T10:06:04.439 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:04.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.562+0000 7fc86edc8640 1 Processor -- start 2024-04-18T10:06:04.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.562+0000 7fc86edc8640 1 -- start start 2024-04-18T10:06:04.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86edc8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc86813fbe0 0x7fc86806bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86edc8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc86813ffc0 0x7fc868149150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86edc8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8680b8750 con 0x7fc86809ce60 2024-04-18T10:06:04.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86edc8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fc8680b89f0 con 0x7fc86813fbe0 2024-04-18T10:06:04.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86edc8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc8680b8cd0 con 0x7fc86813ffc0 2024-04-18T10:06:04.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86cb64640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc86813fbe0 0x7fc86806bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86cb64640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc86813fbe0 0x7fc86806bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53140/0 (socket says 172.21.15.62:53140) 2024-04-18T10:06:04.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86d365640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc86809ce60 0x7fc868145aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54898/0 (socket says 172.21.15.62:54898) 2024-04-18T10:06:04.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86cb64640 1 -- 172.21.15.62:0/2667696916 learned_addr learned my addr 172.21.15.62:0/2667696916 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:04.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86cb64640 1 -- 172.21.15.62:0/2667696916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc86813ffc0 msgr2=0x7fc868149150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:04.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc867fff640 1 --2- 172.21.15.62:0/2667696916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc86813ffc0 0x7fc868149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86cb64640 1 --2- 172.21.15.62:0/2667696916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc86813ffc0 0x7fc868149150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc8677fe640 1 -- 172.21.15.62:0/2667696916 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 990619531 0 0) 0x7fc8680b8750 con 0x7fc86809ce60 2024-04-18T10:06:04.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86cb64640 1 -- 172.21.15.62:0/2667696916 >> v1:172.21.15.62:6789/0 conn(0x7fc86809ce60 legacy=0x7fc868145aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86cb64640 1 -- 172.21.15.62:0/2667696916 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86806c010 con 0x7fc86813fbe0 2024-04-18T10:06:04.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc867fff640 1 --2- 172.21.15.62:0/2667696916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc86813ffc0 0x7fc868149150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:04.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc86cb64640 1 --2- 172.21.15.62:0/2667696916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc86813fbe0 0x7fc86806bad0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc854009870 tx=0x7fc85401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5120d75d44214b server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc8677fe640 1 -- 172.21.15.62:0/2667696916 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8540b1070 con 0x7fc86813fbe0 2024-04-18T10:06:04.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc8677fe640 1 -- 172.21.15.62:0/2667696916 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8540a8050 con 0x7fc86813fbe0 2024-04-18T10:06:04.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.566+0000 7fc8677fe640 1 -- 172.21.15.62:0/2667696916 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8540ac860 con 0x7fc86813fbe0 2024-04-18T10:06:04.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 -- 172.21.15.62:0/2667696916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc86813fbe0 msgr2=0x7fc86806bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 --2- 172.21.15.62:0/2667696916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc86813fbe0 0x7fc86806bad0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc854009870 tx=0x7fc85401c4d0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 -- 172.21.15.62:0/2667696916 shutdown_connections 2024-04-18T10:06:04.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 --2- 172.21.15.62:0/2667696916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc86813ffc0 0x7fc868149150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 --2- 172.21.15.62:0/2667696916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc86813fbe0 0x7fc86806bad0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 -- 172.21.15.62:0/2667696916 >> 172.21.15.62:0/2667696916 conn(0x7fc86809c250 msgr2=0x7fc86809c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:04.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 -- 172.21.15.62:0/2667696916 shutdown_connections 2024-04-18T10:06:04.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 -- 172.21.15.62:0/2667696916 wait complete. 2024-04-18T10:06:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 Processor -- start 2024-04-18T10:06:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 -- start start 2024-04-18T10:06:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc86809ce60 0x7fc8680b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc86813fbe0 0x7fc868094da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc86813ffc0 0x7fc86809a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc868153400 con 0x7fc86809ce60 2024-04-18T10:06:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fc86806f8c0 con 0x7fc86813ffc0 2024-04-18T10:06:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc868153580 con 0x7fc86813fbe0 2024-04-18T10:06:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86cb64640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc86809ce60 0x7fc8680b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86cb64640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc86809ce60 0x7fc8680b05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53150/0 (socket says 172.21.15.62:53150) 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86cb64640 1 -- 172.21.15.62:0/904558666 learned_addr learned my addr 172.21.15.62:0/904558666 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86cb64640 1 -- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc86813fbe0 msgr2=0x7fc868094da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86d365640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc86813ffc0 0x7fc86809a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86cb64640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc86813fbe0 0x7fc868094da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86cb64640 1 -- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc86813ffc0 msgr2=0x7fc86809a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86cb64640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc86813ffc0 0x7fc86809a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86cb64640 1 -- 172.21.15.62:0/904558666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86809a6c0 con 0x7fc86809ce60 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86cb64640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc86809ce60 0x7fc8680b05b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc8540af040 tx=0x7fc85401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc8657fa640 1 -- 172.21.15.62:0/904558666 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8540bb020 con 0x7fc86809ce60 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc8657fa640 1 -- 172.21.15.62:0/904558666 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8540b7420 con 0x7fc86809ce60 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 -- 172.21.15.62:0/904558666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc86809b5b0 con 0x7fc86809ce60 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc8657fa640 1 -- 172.21.15.62:0/904558666 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8540b1040 con 0x7fc86809ce60 2024-04-18T10:06:04.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 -- 172.21.15.62:0/904558666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8680952e0 con 0x7fc86809ce60 2024-04-18T10:06:04.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.570+0000 7fc86edc8640 1 -- 172.21.15.62:0/904558666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc868069230 con 0x7fc86809ce60 2024-04-18T10:06:04.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.574+0000 7fc8657fa640 1 -- 172.21.15.62:0/904558666 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc8540be020 con 0x7fc86809ce60 2024-04-18T10:06:04.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.574+0000 7fc8657fa640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc84c068380 0x7fc84c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.574+0000 7fc8657fa640 1 -- 172.21.15.62:0/904558666 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc854120cd0 con 0x7fc86809ce60 2024-04-18T10:06:04.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.574+0000 7fc867fff640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc84c068380 0x7fc84c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.574+0000 7fc867fff640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc84c068380 0x7fc84c06a840 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc8680b0d20 tx=0x7fc85c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.578+0000 7fc8657fa640 1 -- 172.21.15.62:0/904558666 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc8540c5030 con 0x7fc86809ce60 2024-04-18T10:06:04.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.742+0000 7fc86edc8640 1 -- 172.21.15.62:0/904558666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc868073f90 con 0x7fc86809ce60 2024-04-18T10:06:04.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.742+0000 7fc8657fa640 1 -- 172.21.15.62:0/904558666 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7fc854016150 con 0x7fc86809ce60 2024-04-18T10:06:04.743 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:04.743 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:04.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.746+0000 7fc86edc8640 1 -- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc84c068380 msgr2=0x7fc84c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.746+0000 7fc86edc8640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc84c068380 0x7fc84c06a840 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc8680b0d20 tx=0x7fc85c008040 comp rx=0 tx=0).stop 2024-04-18T10:06:04.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.746+0000 7fc86edc8640 1 -- 172.21.15.62:0/904558666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc86809ce60 msgr2=0x7fc8680b05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.746+0000 7fc86edc8640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc86809ce60 0x7fc8680b05b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc8540af040 tx=0x7fc85401ad10 comp rx=0 tx=0).stop 2024-04-18T10:06:04.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.746+0000 7fc86edc8640 1 -- 172.21.15.62:0/904558666 shutdown_connections 2024-04-18T10:06:04.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.746+0000 7fc86edc8640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc84c068380 0x7fc84c06a840 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.746+0000 7fc86edc8640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc86813ffc0 0x7fc86809a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.746+0000 7fc86edc8640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc86813fbe0 0x7fc868094da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.746+0000 7fc86edc8640 1 --2- 172.21.15.62:0/904558666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc86809ce60 0x7fc8680b05b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.746+0000 7fc86edc8640 1 -- 172.21.15.62:0/904558666 >> 172.21.15.62:0/904558666 conn(0x7fc86809c250 msgr2=0x7fc8680b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:04.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.746+0000 7fc86edc8640 1 -- 172.21.15.62:0/904558666 shutdown_connections 2024-04-18T10:06:04.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.746+0000 7fc86edc8640 1 -- 172.21.15.62:0/904558666 wait complete. 2024-04-18T10:06:04.763 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-18T10:06:04.763 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-18T10:06:04.764 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-18T10:06:04.764 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-18T10:06:04.764 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-18T10:06:04.765 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-18T10:06:04.765 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-18T10:06:04.765 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-18T10:06:04.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06c0c74640 1 Processor -- start 2024-04-18T10:06:04.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06c0c74640 1 -- start start 2024-04-18T10:06:04.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fb01d55e640 1 Processor -- start 2024-04-18T10:06:04.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fe592309640 1 Processor -- start 2024-04-18T10:06:04.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fe592309640 1 -- start start 2024-04-18T10:06:04.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fb01d55e640 1 -- start start 2024-04-18T10:06:04.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06c0c74640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f06bc072d50 0x7f06bc148030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06c0c74640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06bc071f30 0x7f06bc0b3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06c0c74640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06bc0682d0 con 0x7f06bc071490 2024-04-18T10:06:04.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06c0c74640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f06bc0b4320 con 0x7f06bc072d50 2024-04-18T10:06:04.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06c0c74640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f06bc073130 con 0x7f06bc071f30 2024-04-18T10:06:04.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06bad9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f06bc071490 0x7f06bc144a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54914/0 (socket says 172.21.15.62:54914) 2024-04-18T10:06:04.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06bad9d640 1 -- 172.21.15.62:0/3664021331 learned_addr learned my addr 172.21.15.62:0/3664021331 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:04.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fe592309640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe58c1498f0 0x7fe58c13f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fe592309640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe58c13fa40 0x7fe58c1501b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fe592309640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe58c0b3040 con 0x7fe58c0b2c60 2024-04-18T10:06:04.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fe58b7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe58c1498f0 0x7fe58c13f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fe592309640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fe58c070e50 con 0x7fe58c1498f0 2024-04-18T10:06:04.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fe58b7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe58c1498f0 0x7fe58c13f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53164/0 (socket says 172.21.15.62:53164) 2024-04-18T10:06:04.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fe58b7fe640 1 -- 172.21.15.62:0/3039377153 learned_addr learned my addr 172.21.15.62:0/3039377153 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:04.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fe592309640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fe58c070890 con 0x7fe58c13fa40 2024-04-18T10:06:04.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fe5908a6640 1 --2- 172.21.15.62:0/3039377153 >> v2:172.21.15.145:3300/0 conn(0x7fe58c13fa40 0x7fe58c1501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f7cd01c6640 1 Processor -- start 2024-04-18T10:06:04.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f7cd01c6640 1 -- start start 2024-04-18T10:06:04.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f88f292a640 1 Processor -- start 2024-04-18T10:06:04.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f88f292a640 1 -- start start 2024-04-18T10:06:04.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fd64952a640 1 Processor -- start 2024-04-18T10:06:04.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fd64952a640 1 -- start start 2024-04-18T10:06:04.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd64952a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6440b2370 0x7fd6440b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fb01d55e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb01000c360 0x7fb0100d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fb01d55e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb01001f1a0 0x7fb01001e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fb01d55e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb01000c740 con 0x7fb0100d0f80 2024-04-18T10:06:04.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fb01d55e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fb010007bd0 con 0x7fb01001f1a0 2024-04-18T10:06:04.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fb01d55e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fb010008470 con 0x7fb01000c360 2024-04-18T10:06:04.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fb017fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb01000c360 0x7fb0100d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fb017fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb01000c360 0x7fb0100d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.62:41844/0 (socket says 172.21.15.62:41844) 2024-04-18T10:06:04.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fb017fff640 1 -- 172.21.15.62:0/2519127668 learned_addr learned my addr 172.21.15.62:0/2519127668 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:04.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7fb0177fe640 1 --2- 172.21.15.62:0/2519127668 >> v2:172.21.15.62:3300/0 conn(0x7fb01001f1a0 0x7fb01001e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fb016ffd640 1 -- 172.21.15.62:0/2519127668 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1442332385 0 0) 0x7fb01000c740 con 0x7fb0100d0f80 2024-04-18T10:06:04.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fb016ffd640 1 -- 172.21.15.62:0/2519127668 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb008003530 con 0x7fb0100d0f80 2024-04-18T10:06:04.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fb017fff640 1 -- 172.21.15.62:0/2519127668 >> v2:172.21.15.62:3300/0 conn(0x7fb01001f1a0 msgr2=0x7fb01001e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:04.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06b9d9b640 1 --2- 172.21.15.62:0/3664021331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06bc071f30 0x7f06bc0b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06ba59c640 1 --2- 172.21.15.62:0/3664021331 >> v2:172.21.15.145:3300/0 conn(0x7f06bc072d50 0x7f06bc148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06b959a640 1 -- 172.21.15.62:0/3664021331 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3778408109 0 0) 0x7f06bc0682d0 con 0x7f06bc071490 2024-04-18T10:06:04.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06b959a640 1 -- 172.21.15.62:0/3664021331 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06a4003530 con 0x7f06bc071490 2024-04-18T10:06:04.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f715359e640 1 Processor -- start 2024-04-18T10:06:04.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f715359e640 1 -- start start 2024-04-18T10:06:04.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f715359e640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f71540b1ec0 0x7f7154145b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f715359e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71540b22a0 0x7f71541460a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f715359e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71540b0dc0 con 0x7f715414c410 2024-04-18T10:06:04.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f715359e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f715414bf30 con 0x7f71540b22a0 2024-04-18T10:06:04.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f715359e640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f71540b1060 con 0x7f71540b1ec0 2024-04-18T10:06:04.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f189aa0a640 1 Processor -- start 2024-04-18T10:06:04.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f189aa0a640 1 -- start start 2024-04-18T10:06:04.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fe58b7fe640 1 -- 172.21.15.62:0/3039377153 >> v2:172.21.15.145:3300/0 conn(0x7fe58c13fa40 msgr2=0x7fe58c1501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:04.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fe58b7fe640 1 --2- 172.21.15.62:0/3039377153 >> v2:172.21.15.145:3300/0 conn(0x7fe58c13fa40 0x7fe58c1501b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fe58b7fe640 1 -- 172.21.15.62:0/3039377153 >> v1:172.21.15.145:6789/0 conn(0x7fe58c0b2c60 legacy=0x7fe58c1491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd642ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6440b2370 0x7fd6440b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd642ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd6440b2370 0x7fd6440b50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53180/0 (socket says 172.21.15.62:53180) 2024-04-18T10:06:04.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f7152d9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f715414c410 0x7f715414c820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54916/0 (socket says 172.21.15.62:54916) 2024-04-18T10:06:04.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f7152d9d640 1 -- 172.21.15.62:0/2354125546 learned_addr learned my addr 172.21.15.62:0/2354125546 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:04.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fb017fff640 1 --2- 172.21.15.62:0/2519127668 >> v2:172.21.15.62:3300/0 conn(0x7fb01001f1a0 0x7fb01001e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fb017fff640 1 -- 172.21.15.62:0/2519127668 >> v1:172.21.15.145:6789/0 conn(0x7fb0100d0f80 legacy=0x7fb0100d3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fb016ffd640 1 -- 172.21.15.62:0/2519127668 <== mon.1 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 719291748 0 0) 0x7fb008003530 con 0x7fb0100d0f80 2024-04-18T10:06:04.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fb017fff640 1 -- 172.21.15.62:0/2519127668 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb010007e20 con 0x7fb01000c360 2024-04-18T10:06:04.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f7151d9b640 1 --2- 172.21.15.62:0/2354125546 >> v2:172.21.15.62:3300/0 conn(0x7f71540b22a0 0x7f71541460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd6427fc640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fd644073aa0 0x7fd644147100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.62:53616/0 (socket says 172.21.15.62:53616) 2024-04-18T10:06:04.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f189aa0a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f18940b9e10 0x7f18940b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f189aa0a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f18940b1880 0x7f18940b5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f189aa0a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18940b2820 con 0x7f1894073aa0 2024-04-18T10:06:04.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f189aa0a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f18940b0d20 con 0x7f18940b9e10 2024-04-18T10:06:04.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f189aa0a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f18940b1000 con 0x7f18940b1880 2024-04-18T10:06:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fb0177fe640 1 --2- 172.21.15.62:0/2519127668 >> v2:172.21.15.62:3300/0 conn(0x7fb01001f1a0 0x7fb01001e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd642ffd640 1 -- 172.21.15.62:0/3063551602 learned_addr learned my addr 172.21.15.62:0/3063551602 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:04.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd64952a640 1 --2- 172.21.15.62:0/3063551602 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6440b1880 0x7fd6440b5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f715259c640 1 --2- 172.21.15.62:0/2354125546 >> v2:172.21.15.145:3300/0 conn(0x7f71540b1ec0 0x7f7154145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f1893fff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f1894073aa0 0x7f18940b2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.62:53630/0 (socket says 172.21.15.62:53630) 2024-04-18T10:06:04.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f1893fff640 1 -- 172.21.15.62:0/138829658 learned_addr learned my addr 172.21.15.62:0/138829658 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:04.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f18937fe640 1 --2- 172.21.15.62:0/138829658 >> v2:172.21.15.62:3300/0 conn(0x7f18940b9e10 0x7f18940b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f88f292a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88ec0b1ed0 0x7f88ec14c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f88f292a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88ec14be10 con 0x7f88ec145ad0 2024-04-18T10:06:04.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f88f292a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88ec0b13e0 con 0x7f88ec0b3cf0 2024-04-18T10:06:04.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f88f292a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f88ec14cdd0 con 0x7f88ec0b1ed0 2024-04-18T10:06:04.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f715159a640 1 -- 172.21.15.62:0/2354125546 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 978274672 0 0) 0x7f71540b0dc0 con 0x7f715414c410 2024-04-18T10:06:04.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f715159a640 1 -- 172.21.15.62:0/2354125546 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f713c003530 con 0x7f715414c410 2024-04-18T10:06:04.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f7151d9b640 1 -- 172.21.15.62:0/2354125546 >> v2:172.21.15.145:3300/0 conn(0x7f71540b1ec0 msgr2=0x7f7154145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:04.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f7151d9b640 1 --2- 172.21.15.62:0/2354125546 >> v2:172.21.15.145:3300/0 conn(0x7f71540b1ec0 0x7f7154145b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f7151d9b640 1 -- 172.21.15.62:0/2354125546 >> v1:172.21.15.62:6789/0 conn(0x7f715414c410 legacy=0x7f715414c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fe58b7fe640 1 -- 172.21.15.62:0/3039377153 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58c070b70 con 0x7fe58c1498f0 2024-04-18T10:06:04.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f7151d9b640 1 -- 172.21.15.62:0/2354125546 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71540b1340 con 0x7f71540b22a0 2024-04-18T10:06:04.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f88ebfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88ec0b1ed0 0x7f88ec14c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd64952a640 1 -- 172.21.15.62:0/3063551602 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6440b0a40 con 0x7fd644073aa0 2024-04-18T10:06:04.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd64952a640 1 -- 172.21.15.62:0/3063551602 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fd6440b0d20 con 0x7fd6440b2370 2024-04-18T10:06:04.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd64952a640 1 -- 172.21.15.62:0/3063551602 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd6440b1000 con 0x7fd6440b1880 2024-04-18T10:06:04.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f88ebfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f88ec0b1ed0 0x7f88ec14c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53188/0 (socket says 172.21.15.62:53188) 2024-04-18T10:06:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f88ebfff640 1 -- 172.21.15.62:0/4031726693 learned_addr learned my addr 172.21.15.62:0/4031726693 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd641ffb640 1 -- 172.21.15.62:0/3063551602 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3303979076 0 0) 0x7fd6440b0a40 con 0x7fd644073aa0 2024-04-18T10:06:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd641ffb640 1 -- 172.21.15.62:0/3063551602 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd62c003530 con 0x7fd644073aa0 2024-04-18T10:06:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f88ebfff640 1 -- 172.21.15.62:0/4031726693 >> v1:172.21.15.145:6789/0 conn(0x7f88ec0b3cf0 legacy=0x7f88ec0b0cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-18T10:06:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f88ebfff640 1 -- 172.21.15.62:0/4031726693 >> v1:172.21.15.62:6789/0 conn(0x7f88ec145ad0 legacy=0x7f88ec145f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-18T10:06:04.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f88ebfff640 1 -- 172.21.15.62:0/4031726693 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ec0b22b0 con 0x7f88ec0b1ed0 2024-04-18T10:06:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd642ffd640 1 -- 172.21.15.62:0/3063551602 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6440b1880 msgr2=0x7fd6440b5600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-18T10:06:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd642ffd640 1 --2- 172.21.15.62:0/3063551602 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6440b1880 0x7fd6440b5600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd642ffd640 1 -- 172.21.15.62:0/3063551602 >> v1:172.21.15.145:6789/0 conn(0x7fd644073aa0 legacy=0x7fd644147100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fd642ffd640 1 -- 172.21.15.62:0/3063551602 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6440b2750 con 0x7fd6440b2370 2024-04-18T10:06:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f18937fe640 1 -- 172.21.15.62:0/138829658 >> v2:172.21.15.145:3300/0 conn(0x7f18940b1880 msgr2=0x7f18940b5600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-18T10:06:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f18937fe640 1 --2- 172.21.15.62:0/138829658 >> v2:172.21.15.145:3300/0 conn(0x7f18940b1880 0x7f18940b5600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f18937fe640 1 -- 172.21.15.62:0/138829658 >> v1:172.21.15.145:6789/0 conn(0x7f1894073aa0 legacy=0x7f18940b2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f18937fe640 1 -- 172.21.15.62:0/138829658 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18940b1510 con 0x7f18940b9e10 2024-04-18T10:06:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06b9d9b640 1 -- 172.21.15.62:0/3664021331 >> v2:172.21.15.145:3300/0 conn(0x7f06bc072d50 msgr2=0x7f06bc148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06b9d9b640 1 --2- 172.21.15.62:0/3664021331 >> v2:172.21.15.145:3300/0 conn(0x7f06bc072d50 0x7f06bc148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06b9d9b640 1 -- 172.21.15.62:0/3664021331 >> v1:172.21.15.62:6789/0 conn(0x7f06bc071490 legacy=0x7f06bc144a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.950+0000 7f06b9d9b640 1 -- 172.21.15.62:0/3664021331 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06bc0b3fd0 con 0x7f06bc071f30 2024-04-18T10:06:04.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f06b9d9b640 1 --2- 172.21.15.62:0/3664021331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06bc071f30 0x7f06bc0b3230 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0097c0 tx=0x7f06ac01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc27c966df934ead server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fe58b7fe640 1 --2- 172.21.15.62:0/3039377153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c1498f0 0x7fe58c13f500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe58c145830 tx=0x7fe58401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0796c2fa5a1e18 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fe58affd640 1 -- 172.21.15.62:0/3039377153 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5840a5840 con 0x7fe58c1498f0 2024-04-18T10:06:04.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fe58affd640 1 -- 172.21.15.62:0/3039377153 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5840a59e0 con 0x7fe58c1498f0 2024-04-18T10:06:04.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7fe58affd640 1 -- 172.21.15.62:0/3039377153 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5840a5db0 con 0x7fe58c1498f0 2024-04-18T10:06:04.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f06b959a640 1 -- 172.21.15.62:0/3664021331 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac0b2070 con 0x7f06bc071f30 2024-04-18T10:06:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f06b959a640 1 -- 172.21.15.62:0/3664021331 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06ac0a9070 con 0x7f06bc071f30 2024-04-18T10:06:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.954+0000 7f06b959a640 1 -- 172.21.15.62:0/3664021331 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac0ad8c0 con 0x7f06bc071f30 2024-04-18T10:06:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7cd01c6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7cc0039d30 0x7f7cc0035480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7cd01c6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7cc0034370 0x7f7cc00359c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7cd01c6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc00389d0 con 0x7f7cc00385f0 2024-04-18T10:06:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7cd01c6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f7cc0034e10 con 0x7f7cc0039d30 2024-04-18T10:06:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7cd01c6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc00348e0 con 0x7f7cc0034370 2024-04-18T10:06:04.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7cce763640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7cc0034370 0x7f7cc00359c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7cce763640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7cc0034370 0x7f7cc00359c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:47784/0 (socket says 172.21.15.62:47784) 2024-04-18T10:06:04.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7cce763640 1 -- 172.21.15.62:0/2426526177 learned_addr learned my addr 172.21.15.62:0/2426526177 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:04.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7ccd761640 1 --2- 172.21.15.62:0/2426526177 >> v2:172.21.15.145:3300/0 conn(0x7f7cc0039d30 0x7f7cc0035480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 -- 172.21.15.62:0/3039377153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c1498f0 msgr2=0x7fe58c13f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 --2- 172.21.15.62:0/3039377153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c1498f0 0x7fe58c13f500 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe58c145830 tx=0x7fe58401c960 comp rx=0 tx=0).stop 2024-04-18T10:06:04.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb017fff640 1 --2- 172.21.15.62:0/2519127668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb01000c360 0x7fb0100d0a40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb010007620 tx=0x7fb00c01c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f33e051f0cb2c23 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 -- 172.21.15.62:0/3664021331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06bc071f30 msgr2=0x7f06bc0b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 --2- 172.21.15.62:0/3664021331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06bc071f30 0x7f06bc0b3230 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0097c0 tx=0x7f06ac01c450 comp rx=0 tx=0).stop 2024-04-18T10:06:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 -- 172.21.15.62:0/3039377153 shutdown_connections 2024-04-18T10:06:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 --2- 172.21.15.62:0/3039377153 >> v2:172.21.15.145:3300/0 conn(0x7fe58c13fa40 0x7fe58c1501b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 --2- 172.21.15.62:0/3039377153 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c1498f0 0x7fe58c13f500 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 -- 172.21.15.62:0/3039377153 >> 172.21.15.62:0/3039377153 conn(0x7fe58c0b92f0 msgr2=0x7fe58c0b9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 -- 172.21.15.62:0/3039377153 shutdown_connections 2024-04-18T10:06:04.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 -- 172.21.15.62:0/3039377153 wait complete. 2024-04-18T10:06:04.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb016ffd640 1 -- 172.21.15.62:0/2519127668 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0b0070 con 0x7fb01000c360 2024-04-18T10:06:04.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 -- 172.21.15.62:0/3664021331 shutdown_connections 2024-04-18T10:06:04.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 --2- 172.21.15.62:0/3664021331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06bc071f30 0x7f06bc0b3230 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 --2- 172.21.15.62:0/3664021331 >> v2:172.21.15.145:3300/0 conn(0x7f06bc072d50 0x7f06bc148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 -- 172.21.15.62:0/3664021331 >> 172.21.15.62:0/3664021331 conn(0x7f06bc0ba270 msgr2=0x7f06bc0ba650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:04.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 -- 172.21.15.62:0/3664021331 shutdown_connections 2024-04-18T10:06:04.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb016ffd640 1 -- 172.21.15.62:0/2519127668 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00c0a4e50 con 0x7fb01000c360 2024-04-18T10:06:04.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb016ffd640 1 -- 172.21.15.62:0/2519127668 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb004001680 con 0x7fb01000c360 2024-04-18T10:06:04.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7ccd761640 1 -- 172.21.15.62:0/2426526177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7cc0034370 msgr2=0x7f7cc00359c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:04.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7ccd761640 1 --2- 172.21.15.62:0/2426526177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7cc0034370 0x7f7cc00359c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7ccd761640 1 -- 172.21.15.62:0/2426526177 >> v1:172.21.15.145:6789/0 conn(0x7f7cc00385f0 legacy=0x7f7cc00c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7ccd761640 1 -- 172.21.15.62:0/2426526177 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc00c93f0 con 0x7f7cc0039d30 2024-04-18T10:06:04.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fd642ffd640 1 --2- 172.21.15.62:0/3063551602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6440b2370 0x7fd6440b50c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd638007360 tx=0x7fd63801e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51f1f44280292d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 -- 172.21.15.62:0/3664021331 wait complete. 2024-04-18T10:06:04.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f7151d9b640 1 --2- 172.21.15.62:0/2354125546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71540b22a0 0x7f71541460a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f71540ba1f0 tx=0x7f714c01c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be732093f716ca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fd641ffb640 1 -- 172.21.15.62:0/3063551602 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638008be0 con 0x7fd6440b2370 2024-04-18T10:06:04.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f18937fe640 1 --2- 172.21.15.62:0/138829658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18940b9e10 0x7f18940b50c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1894144c10 tx=0x7f188401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1dcd44b6a812d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fd641ffb640 1 -- 172.21.15.62:0/3063551602 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd638008d80 con 0x7fd6440b2370 2024-04-18T10:06:04.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fd641ffb640 1 -- 172.21.15.62:0/3063551602 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638008be0 con 0x7fd6440b2370 2024-04-18T10:06:04.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f715159a640 1 -- 172.21.15.62:0/2354125546 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714c0b2070 con 0x7f71540b22a0 2024-04-18T10:06:04.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f715159a640 1 -- 172.21.15.62:0/2354125546 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f714c0a5770 con 0x7f71540b22a0 2024-04-18T10:06:04.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb01d55e640 1 -- 172.21.15.62:0/2519127668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb01000c360 msgr2=0x7fb0100d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb01d55e640 1 --2- 172.21.15.62:0/2519127668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb01000c360 0x7fb0100d0a40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb010007620 tx=0x7fb00c01c440 comp rx=0 tx=0).stop 2024-04-18T10:06:04.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb01d55e640 1 -- 172.21.15.62:0/2519127668 shutdown_connections 2024-04-18T10:06:04.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb01d55e640 1 --2- 172.21.15.62:0/2519127668 >> v2:172.21.15.62:3300/0 conn(0x7fb01001f1a0 0x7fb01001e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb01d55e640 1 --2- 172.21.15.62:0/2519127668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb01000c360 0x7fb0100d0a40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb01d55e640 1 -- 172.21.15.62:0/2519127668 >> 172.21.15.62:0/2519127668 conn(0x7fb01000b5e0 msgr2=0x7fb01000b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:04.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f88ebfff640 1 --2- 172.21.15.62:0/4031726693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88ec0b1ed0 0x7f88ec14c890 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009e50 tx=0x7f88e401c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f4fa48d4e41cb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb01d55e640 1 -- 172.21.15.62:0/2519127668 shutdown_connections 2024-04-18T10:06:04.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 Processor -- start 2024-04-18T10:06:04.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 -- start start 2024-04-18T10:06:04.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f88eaffd640 1 -- 172.21.15.62:0/4031726693 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40a9480 con 0x7f88ec0b1ed0 2024-04-18T10:06:04.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f88eaffd640 1 -- 172.21.15.62:0/4031726693 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e40ae070 con 0x7f88ec0b1ed0 2024-04-18T10:06:04.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f88eaffd640 1 -- 172.21.15.62:0/4031726693 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40a9840 con 0x7f88ec0b1ed0 2024-04-18T10:06:04.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fd64952a640 1 -- 172.21.15.62:0/3063551602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6440b2370 msgr2=0x7fd6440b50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fd64952a640 1 --2- 172.21.15.62:0/3063551602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6440b2370 0x7fd6440b50c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd638007360 tx=0x7fd63801e5f0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fd64952a640 1 -- 172.21.15.62:0/3063551602 shutdown_connections 2024-04-18T10:06:04.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fd64952a640 1 --2- 172.21.15.62:0/3063551602 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6440b1880 0x7fd6440b5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fd64952a640 1 --2- 172.21.15.62:0/3063551602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6440b2370 0x7fd6440b50c0 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd638007360 tx=0x7fd63801e5f0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fd64952a640 1 -- 172.21.15.62:0/3063551602 >> 172.21.15.62:0/3063551602 conn(0x7fd6440b92f0 msgr2=0x7fd6440b9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:04.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb01d55e640 1 -- 172.21.15.62:0/2519127668 wait complete. 2024-04-18T10:06:04.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 Processor -- start 2024-04-18T10:06:04.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 -- start start 2024-04-18T10:06:04.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f715159a640 1 -- 172.21.15.62:0/2354125546 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714c0a5b40 con 0x7f71540b22a0 2024-04-18T10:06:04.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f715359e640 1 -- 172.21.15.62:0/2354125546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71540b22a0 msgr2=0x7f71541460a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f715359e640 1 --2- 172.21.15.62:0/2354125546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71540b22a0 0x7f71541460a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f71540ba1f0 tx=0x7f714c01c860 comp rx=0 tx=0).stop 2024-04-18T10:06:04.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f1892ffd640 1 -- 172.21.15.62:0/138829658 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18840b0070 con 0x7f18940b9e10 2024-04-18T10:06:04.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f1892ffd640 1 -- 172.21.15.62:0/138829658 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18840ab600 con 0x7f18940b9e10 2024-04-18T10:06:04.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f1892ffd640 1 -- 172.21.15.62:0/138829658 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18840ab9d0 con 0x7f18940b9e10 2024-04-18T10:06:04.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06bc071490 0x7f06bc13f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06bc071f30 0x7f06bc13fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06bc072d50 0x7f06bc13cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f06bc06bc50 con 0x7f06bc071f30 2024-04-18T10:06:04.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f06bc06bad0 con 0x7f06bc072d50 2024-04-18T10:06:04.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06c0c74640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f06bc06bdd0 con 0x7f06bc071490 2024-04-18T10:06:04.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c0b2c60 0x7fe58c142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe58c13fa40 0x7fe58c13abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f88f292a640 1 -- 172.21.15.62:0/4031726693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88ec0b1ed0 msgr2=0x7f88ec14c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f88f292a640 1 --2- 172.21.15.62:0/4031726693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88ec0b1ed0 0x7f88ec14c890 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009e50 tx=0x7f88e401c6b0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb01d55e640 1 Processor -- start 2024-04-18T10:06:04.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe58c1498f0 0x7fe58c1401a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:04.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c0b5090 con 0x7fe58c0b2c60 2024-04-18T10:06:04.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c0b4f10 con 0x7fe58c13fa40 2024-04-18T10:06:04.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fe592309640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fe58c0704b0 con 0x7fe58c1498f0 2024-04-18T10:06:04.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f189aa0a640 1 -- 172.21.15.62:0/138829658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18940b9e10 msgr2=0x7f18940b50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f189aa0a640 1 --2- 172.21.15.62:0/138829658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18940b9e10 0x7f18940b50c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1894144c10 tx=0x7f188401c410 comp rx=0 tx=0).stop 2024-04-18T10:06:04.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fd64952a640 1 -- 172.21.15.62:0/3063551602 shutdown_connections 2024-04-18T10:06:04.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f88f292a640 1 -- 172.21.15.62:0/4031726693 shutdown_connections 2024-04-18T10:06:04.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f88f292a640 1 --2- 172.21.15.62:0/4031726693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88ec0b1ed0 0x7f88ec14c890 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f88f292a640 1 -- 172.21.15.62:0/4031726693 >> 172.21.15.62:0/4031726693 conn(0x7f88ec0b92f0 msgr2=0x7f88ec0b9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:04.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f715359e640 1 -- 172.21.15.62:0/2354125546 shutdown_connections 2024-04-18T10:06:04.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f715359e640 1 --2- 172.21.15.62:0/2354125546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71540b22a0 0x7f71541460a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f715359e640 1 --2- 172.21.15.62:0/2354125546 >> v2:172.21.15.145:3300/0 conn(0x7f71540b1ec0 0x7f7154145b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f715359e640 1 -- 172.21.15.62:0/2354125546 >> 172.21.15.62:0/2354125546 conn(0x7f71540b91f0 msgr2=0x7f71540b9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:04.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f88f292a640 1 -- 172.21.15.62:0/4031726693 shutdown_connections 2024-04-18T10:06:04.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f88f292a640 1 -- 172.21.15.62:0/4031726693 wait complete. 2024-04-18T10:06:04.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7fb01d55e640 1 -- start start 2024-04-18T10:06:04.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06ba59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06bc071490 0x7f06bc13f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.991 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.958+0000 7f06ba59c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06bc071490 0x7f06bc13f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:47790/0 (socket says 172.21.15.62:47790) 2024-04-18T10:06:04.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f06ba59c640 1 -- 172.21.15.62:0/1303004177 learned_addr learned my addr 172.21.15.62:0/1303004177 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:04.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f715359e640 1 -- 172.21.15.62:0/2354125546 shutdown_connections 2024-04-18T10:06:04.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 -- 172.21.15.62:0/138829658 shutdown_connections 2024-04-18T10:06:04.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 --2- 172.21.15.62:0/138829658 >> v2:172.21.15.145:3300/0 conn(0x7f18940b1880 0x7f18940b5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 --2- 172.21.15.62:0/138829658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18940b9e10 0x7f18940b50c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 -- 172.21.15.62:0/138829658 >> 172.21.15.62:0/138829658 conn(0x7f18940b92f0 msgr2=0x7f18940b9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:04.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f06bad9d640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06bc072d50 0x7f06bc13cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f06b9d9b640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06bc071f30 0x7f06bc13fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe58bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c0b2c60 0x7fe58c142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 -- 172.21.15.62:0/138829658 shutdown_connections 2024-04-18T10:06:04.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe58bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c0b2c60 0x7fe58c142de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53210/0 (socket says 172.21.15.62:53210) 2024-04-18T10:06:04.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe58bfff640 1 -- 172.21.15.62:0/3025234477 learned_addr learned my addr 172.21.15.62:0/3025234477 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:04.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe5908a6640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe58c1498f0 0x7fe58c1401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe58bfff640 1 -- 172.21.15.62:0/3025234477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe58c1498f0 msgr2=0x7fe58c1401a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe58bfff640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe58c1498f0 0x7fe58c1401a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe58bfff640 1 -- 172.21.15.62:0/3025234477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe58c13fa40 msgr2=0x7fe58c13abe0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-18T10:06:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe58bfff640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe58c13fa40 0x7fe58c13abe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:04.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe58bfff640 1 -- 172.21.15.62:0/3025234477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58c13b120 con 0x7fe58c0b2c60 2024-04-18T10:06:04.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f88f292a640 1 Processor -- start 2024-04-18T10:06:04.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f715359e640 1 -- 172.21.15.62:0/2354125546 wait complete. 2024-04-18T10:06:04.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe58bfff640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c0b2c60 0x7fe58c142de0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00c3f0 tx=0x7fe57c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:04.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f715359e640 1 Processor -- start 2024-04-18T10:06:05.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f88f292a640 1 -- start start 2024-04-18T10:06:05.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe588ff9640 1 -- 172.21.15.62:0/3025234477 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c017070 con 0x7fe58c0b2c60 2024-04-18T10:06:05.000 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe588ff9640 1 -- 172.21.15.62:0/3025234477 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe57c0052d0 con 0x7fe58c0b2c60 2024-04-18T10:06:05.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe588ff9640 1 -- 172.21.15.62:0/3025234477 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c003840 con 0x7fe58c0b2c60 2024-04-18T10:06:05.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fd64952a640 1 -- 172.21.15.62:0/3063551602 wait complete. 2024-04-18T10:06:05.001 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fd64952a640 1 Processor -- start 2024-04-18T10:06:05.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 -- 172.21.15.62:0/138829658 wait complete. 2024-04-18T10:06:05.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 Processor -- start 2024-04-18T10:06:05.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7ccd761640 1 --2- 172.21.15.62:0/2426526177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7cc0039d30 0x7f7cc0035480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80097c0 tx=0x7f7cb801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b2259d9abea4d91 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb01d55e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb01000c360 0x7fb010035da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb01d55e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb01001f1a0 0x7fb010172110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb01d55e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0100d0f80 0x7fb010172650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb01d55e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb0100de4c0 con 0x7fb0100d0f80 2024-04-18T10:06:05.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb01d55e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fb0100de340 con 0x7fb01000c360 2024-04-18T10:06:05.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb01d55e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fb0100de640 con 0x7fb01001f1a0 2024-04-18T10:06:05.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb0177fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb01001f1a0 0x7fb010172110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb0177fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb01001f1a0 0x7fb010172110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:47810/0 (socket says 172.21.15.62:47810) 2024-04-18T10:06:05.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb0177fe640 1 -- 172.21.15.62:0/2957968992 learned_addr learned my addr 172.21.15.62:0/2957968992 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb017fff640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb01000c360 0x7fb010035da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb01cd5d640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0100d0f80 0x7fb010172650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe592309640 1 -- 172.21.15.62:0/3025234477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe58c13b410 con 0x7fe58c0b2c60 2024-04-18T10:06:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fe592309640 1 -- 172.21.15.62:0/3025234477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe58c145e30 con 0x7fe58c0b2c60 2024-04-18T10:06:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f715359e640 1 -- start start 2024-04-18T10:06:05.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f715359e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71540b1ec0 0x7f71541378d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f715359e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71540b22a0 0x7f7154146950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f715359e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f715414c410 0x7f7154146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f715359e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7154157340 con 0x7f71540b22a0 2024-04-18T10:06:05.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f715359e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f71541571c0 con 0x7f715414c410 2024-04-18T10:06:05.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f715359e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f71541574c0 con 0x7f71540b1ec0 2024-04-18T10:06:05.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f06b9d9b640 1 -- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06bc071490 msgr2=0x7f06bc13f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f06b9d9b640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06bc071490 0x7f06bc13f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f06b9d9b640 1 -- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06bc072d50 msgr2=0x7f06bc13cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f06b9d9b640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06bc072d50 0x7f06bc13cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f06b9d9b640 1 -- 172.21.15.62:0/1303004177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06bc1e5100 con 0x7f06bc071f30 2024-04-18T10:06:05.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f06b9d9b640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06bc071f30 0x7f06bc13fd00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f06bc0b44a0 tx=0x7f06ac0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7152d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f715414c410 0x7f7154146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7152d9d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f715414c410 0x7f7154146e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.62:41894/0 (socket says 172.21.15.62:41894) 2024-04-18T10:06:05.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7152d9d640 1 -- 172.21.15.62:0/2875069765 learned_addr learned my addr 172.21.15.62:0/2875069765 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7cccf60640 1 -- 172.21.15.62:0/2426526177 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80b0070 con 0x7f7cc0039d30 2024-04-18T10:06:05.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7cccf60640 1 -- 172.21.15.62:0/2426526177 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cb80ab600 con 0x7f7cc0039d30 2024-04-18T10:06:05.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7cccf60640 1 -- 172.21.15.62:0/2426526177 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80ab9d0 con 0x7f7cc0039d30 2024-04-18T10:06:05.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7cd01c6640 1 -- 172.21.15.62:0/2426526177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7cc0039d30 msgr2=0x7f7cc0035480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7cd01c6640 1 --2- 172.21.15.62:0/2426526177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7cc0039d30 0x7f7cc0035480 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80097c0 tx=0x7f7cb801c410 comp rx=0 tx=0).stop 2024-04-18T10:06:05.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 -- start start 2024-04-18T10:06:05.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1894073aa0 0x7f1894141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18940b1880 0x7f1894133ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18940b9e10 0x7f18941340e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f189414afa0 con 0x7f1894073aa0 2024-04-18T10:06:05.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f189414ae20 con 0x7f18940b1880 2024-04-18T10:06:05.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f189aa0a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f189414b120 con 0x7f18940b9e10 2024-04-18T10:06:05.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f1898fa7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18940b9e10 0x7f18941340e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f88f292a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88ec0b1ed0 0x7f88ec149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f88f292a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88ec0b3cf0 0x7f88ec1e4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f88f292a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88ec145ad0 0x7f88ec1e51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f88f292a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec0b5c40 con 0x7f88ec0b1ed0 2024-04-18T10:06:05.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f88f292a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f88ec14c400 con 0x7f88ec0b3cf0 2024-04-18T10:06:05.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f88f292a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f88ec15a1d0 con 0x7f88ec145ad0 2024-04-18T10:06:05.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7cd01c6640 1 -- 172.21.15.62:0/2426526177 shutdown_connections 2024-04-18T10:06:05.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7cd01c6640 1 --2- 172.21.15.62:0/2426526177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7cc0034370 0x7f7cc00359c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7cd01c6640 1 --2- 172.21.15.62:0/2426526177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7cc0039d30 0x7f7cc0035480 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7cd01c6640 1 -- 172.21.15.62:0/2426526177 >> 172.21.15.62:0/2426526177 conn(0x7f7cc003abd0 msgr2=0x7f7cc0005ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f715259c640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71540b1ec0 0x7f71541378d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7cd01c6640 1 -- 172.21.15.62:0/2426526177 shutdown_connections 2024-04-18T10:06:05.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fd64952a640 1 -- start start 2024-04-18T10:06:05.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fd64952a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd644073aa0 0x7fd6441e4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fd64952a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6440b1880 0x7fd6441e4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fd64952a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd644149e10 0x7fd6441f0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fd64952a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd644150f90 con 0x7fd644149e10 2024-04-18T10:06:05.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fd64952a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fd644150e10 con 0x7fd6440b1880 2024-04-18T10:06:05.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fd64952a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd644151110 con 0x7fd644073aa0 2024-04-18T10:06:05.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f88f0ec7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88ec145ad0 0x7f88ec1e51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb0177fe640 1 -- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb01000c360 msgr2=0x7fb010035da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb0177fe640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb01000c360 0x7fb010035da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb0177fe640 1 -- 172.21.15.62:0/2957968992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0100d0f80 msgr2=0x7fb010172650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb0177fe640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0100d0f80 0x7fb010172650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7fb0177fe640 1 -- 172.21.15.62:0/2957968992 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb010026ad0 con 0x7fb01001f1a0 2024-04-18T10:06:05.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f1898fa7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18940b9e10 0x7f18941340e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:47818/0 (socket says 172.21.15.62:47818) 2024-04-18T10:06:05.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f1898fa7640 1 -- 172.21.15.62:0/3650078111 learned_addr learned my addr 172.21.15.62:0/3650078111 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f18937fe640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18940b1880 0x7f1894133ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88f0ec7640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88ec145ad0 0x7f88ec1e51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:47822/0 (socket says 172.21.15.62:47822) 2024-04-18T10:06:05.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88f0ec7640 1 -- 172.21.15.62:0/1143537152 learned_addr learned my addr 172.21.15.62:0/1143537152 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88eb7fe640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88ec0b3cf0 0x7f88ec1e4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd642ffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd644073aa0 0x7fd6441e4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd642ffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd644073aa0 0x7fd6441e4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:47832/0 (socket says 172.21.15.62:47832) 2024-04-18T10:06:05.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd642ffd640 1 -- 172.21.15.62:0/50474965 learned_addr learned my addr 172.21.15.62:0/50474965 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fb0177fe640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb01001f1a0 0x7fb010172110 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb01806a910 tx=0x7fb018069600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.962+0000 7f7cd01c6640 1 -- 172.21.15.62:0/2426526177 wait complete. 2024-04-18T10:06:05.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd6427fc640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6440b1880 0x7fd6441e4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7152d9d640 1 -- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71540b1ec0 msgr2=0x7f71541378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7152d9d640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71540b1ec0 0x7f71541378d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7152d9d640 1 -- 172.21.15.62:0/2875069765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71540b22a0 msgr2=0x7f7154146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7152d9d640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71540b22a0 0x7f7154146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.024 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7152d9d640 1 -- 172.21.15.62:0/2875069765 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7154134200 con 0x7f715414c410 2024-04-18T10:06:05.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f1898fa7640 1 -- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18940b1880 msgr2=0x7f1894133ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f1898fa7640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18940b1880 0x7f1894133ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f1898fa7640 1 -- 172.21.15.62:0/3650078111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1894073aa0 msgr2=0x7f1894141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f1898fa7640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1894073aa0 0x7f1894141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f1898fa7640 1 -- 172.21.15.62:0/3650078111 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1894132a70 con 0x7f18940b9e10 2024-04-18T10:06:05.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f1898fa7640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18940b9e10 0x7f18941340e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f188800b410 tx=0x7f188800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fb014ff9640 1 -- 172.21.15.62:0/2957968992 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb018072ab0 con 0x7fb01001f1a0 2024-04-18T10:06:05.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7152d9d640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f715414c410 0x7f7154146e90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f714400c390 tx=0x7f714400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fb01d55e640 1 -- 172.21.15.62:0/2957968992 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb010027be0 con 0x7fb01001f1a0 2024-04-18T10:06:05.027 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88ebfff640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88ec0b1ed0 0x7f88ec149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7cd01c6640 1 Processor -- start 2024-04-18T10:06:05.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd6437fe640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd644149e10 0x7fd6441f0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd642ffd640 1 -- 172.21.15.62:0/50474965 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6440b1880 msgr2=0x7fd6441e4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd642ffd640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6440b1880 0x7fd6441e4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd642ffd640 1 -- 172.21.15.62:0/50474965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd644149e10 msgr2=0x7fd6441f0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd642ffd640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd644149e10 0x7fd6441f0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd642ffd640 1 -- 172.21.15.62:0/50474965 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6441e4ec0 con 0x7fd644073aa0 2024-04-18T10:06:05.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fb01d55e640 1 -- 172.21.15.62:0/2957968992 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb010027900 con 0x7fb01001f1a0 2024-04-18T10:06:05.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fb014ff9640 1 -- 172.21.15.62:0/2957968992 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb018072c50 con 0x7fb01001f1a0 2024-04-18T10:06:05.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fb014ff9640 1 -- 172.21.15.62:0/2957968992 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb018094480 con 0x7fb01001f1a0 2024-04-18T10:06:05.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f06aaffd640 1 -- 172.21.15.62:0/1303004177 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac0b2040 con 0x7f06bc071f30 2024-04-18T10:06:05.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f06aaffd640 1 -- 172.21.15.62:0/1303004177 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06ac0bb070 con 0x7f06bc071f30 2024-04-18T10:06:05.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f06aaffd640 1 -- 172.21.15.62:0/1303004177 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac0b6400 con 0x7f06bc071f30 2024-04-18T10:06:05.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88ebfff640 1 -- 172.21.15.62:0/1143537152 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88ec145ad0 msgr2=0x7f88ec1e51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88ebfff640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88ec145ad0 0x7f88ec1e51d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd642ffd640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd644073aa0 0x7fd6441e4440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd63801cca0 tx=0x7fd63801b390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88ebfff640 1 -- 172.21.15.62:0/1143537152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88ec0b3cf0 msgr2=0x7f88ec1e4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88ebfff640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88ec0b3cf0 0x7f88ec1e4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88ebfff640 1 -- 172.21.15.62:0/1143537152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ec148b10 con 0x7f88ec0b1ed0 2024-04-18T10:06:05.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88ebfff640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88ec0b1ed0 0x7f88ec149500 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f88e40aa000 tx=0x7f88e4019450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fb01d55e640 1 -- 172.21.15.62:0/2957968992 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faff8001040 con 0x7fb01001f1a0 2024-04-18T10:06:05.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7142ffd640 1 -- 172.21.15.62:0/2875069765 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71440074e0 con 0x7f715414c410 2024-04-18T10:06:05.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7142ffd640 1 -- 172.21.15.62:0/2875069765 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71440047f0 con 0x7f715414c410 2024-04-18T10:06:05.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7142ffd640 1 -- 172.21.15.62:0/2875069765 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144007680 con 0x7f715414c410 2024-04-18T10:06:05.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7cd01c6640 1 -- start start 2024-04-18T10:06:05.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7cd01c6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cc0034370 0x7f7cc00c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7cd01c6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7cc00385f0 0x7f7cc00c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7cd01c6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7cc00b8d00 0x7f7cc00c7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7cd01c6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc00d4700 con 0x7f7cc0034370 2024-04-18T10:06:05.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7cd01c6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc00d4580 con 0x7f7cc00b8d00 2024-04-18T10:06:05.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7cd01c6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc00d4880 con 0x7f7cc00385f0 2024-04-18T10:06:05.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7ccdf62640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cc0034370 0x7f7cc00c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7ccdf62640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cc0034370 0x7f7cc00c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53266/0 (socket says 172.21.15.62:53266) 2024-04-18T10:06:05.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7ccdf62640 1 -- 172.21.15.62:0/2691087558 learned_addr learned my addr 172.21.15.62:0/2691087558 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88e8ff9640 1 -- 172.21.15.62:0/1143537152 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40b8070 con 0x7f88ec0b1ed0 2024-04-18T10:06:05.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f715359e640 1 -- 172.21.15.62:0/2875069765 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7154134750 con 0x7f715414c410 2024-04-18T10:06:05.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f1890ff9640 1 -- 172.21.15.62:0/3650078111 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1888015020 con 0x7f18940b9e10 2024-04-18T10:06:05.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f1890ff9640 1 -- 172.21.15.62:0/3650078111 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1888011830 con 0x7f18940b9e10 2024-04-18T10:06:05.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f1890ff9640 1 -- 172.21.15.62:0/3650078111 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1888011c20 con 0x7f18940b9e10 2024-04-18T10:06:05.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f189aa0a640 1 -- 172.21.15.62:0/3650078111 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1894132ce0 con 0x7f18940b9e10 2024-04-18T10:06:05.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f189aa0a640 1 -- 172.21.15.62:0/3650078111 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f18940b2110 con 0x7f18940b9e10 2024-04-18T10:06:05.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f715359e640 1 -- 172.21.15.62:0/2875069765 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7154133570 con 0x7f715414c410 2024-04-18T10:06:05.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7cce763640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7cc00b8d00 0x7f7cc00c7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd62b7fe640 1 -- 172.21.15.62:0/50474965 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638008be0 con 0x7fd644073aa0 2024-04-18T10:06:05.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd64952a640 1 -- 172.21.15.62:0/50474965 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd644149660 con 0x7fd644073aa0 2024-04-18T10:06:05.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7fd64952a640 1 -- 172.21.15.62:0/50474965 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd64414b060 con 0x7fd644073aa0 2024-04-18T10:06:05.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7ccdf62640 1 -- 172.21.15.62:0/2691087558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7cc00385f0 msgr2=0x7f7cc00c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7ccdf62640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7cc00385f0 0x7f7cc00c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7ccdf62640 1 -- 172.21.15.62:0/2691087558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7cc00b8d00 msgr2=0x7f7cc00c7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7ccdf62640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7cc00b8d00 0x7f7cc00c7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7ccdf62640 1 -- 172.21.15.62:0/2691087558 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc00c4010 con 0x7f7cc0034370 2024-04-18T10:06:05.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7ccdf62640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cc0034370 0x7f7cc00c8dc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400c500 tx=0x7f7cc400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f715359e640 1 -- 172.21.15.62:0/2875069765 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f715404fe10 con 0x7f715414c410 2024-04-18T10:06:05.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88e8ff9640 1 -- 172.21.15.62:0/1143537152 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e40a9a30 con 0x7f88ec0b1ed0 2024-04-18T10:06:05.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88e8ff9640 1 -- 172.21.15.62:0/1143537152 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40a9e20 con 0x7f88ec0b1ed0 2024-04-18T10:06:05.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88f292a640 1 -- 172.21.15.62:0/1143537152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88ec145830 con 0x7f88ec0b1ed0 2024-04-18T10:06:05.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88f292a640 1 -- 172.21.15.62:0/1143537152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88ec145270 con 0x7f88ec0b1ed0 2024-04-18T10:06:05.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f88f292a640 1 -- 172.21.15.62:0/1143537152 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f88ec04fe10 con 0x7f88ec0b1ed0 2024-04-18T10:06:05.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f189aa0a640 1 -- 172.21.15.62:0/3650078111 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f189404fe10 con 0x7f18940b9e10 2024-04-18T10:06:05.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fe592309640 1 -- 172.21.15.62:0/3025234477 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe56c001040 con 0x7fe58c0b2c60 2024-04-18T10:06:05.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fe588ff9640 1 -- 172.21.15.62:0/3025234477 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe57c01a070 con 0x7fe58c0b2c60 2024-04-18T10:06:05.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7cbe7fc640 1 -- 172.21.15.62:0/2691087558 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4016020 con 0x7f7cc0034370 2024-04-18T10:06:05.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f7cd01c6640 1 -- 172.21.15.62:0/2691087558 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc00bc840 con 0x7f7cc0034370 2024-04-18T10:06:05.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f7cd01c6640 1 -- 172.21.15.62:0/2691087558 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc00b9790 con 0x7f7cc0034370 2024-04-18T10:06:05.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fd64952a640 1 -- 172.21.15.62:0/50474965 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd64404fe10 con 0x7fd644073aa0 2024-04-18T10:06:05.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fd62b7fe640 1 -- 172.21.15.62:0/50474965 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd638008d80 con 0x7fd644073aa0 2024-04-18T10:06:05.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fe588ff9640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe578068380 0x7fe57806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fe588ff9640 1 -- 172.21.15.62:0/3025234477 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe57c085c70 con 0x7fe58c0b2c60 2024-04-18T10:06:05.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fe588ff9640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] conn(0x7fe578070580 0x7fe5780729a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fe588ff9640 1 -- 172.21.15.62:0/3025234477 --> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe57c016ec0 con 0x7fe578070580 2024-04-18T10:06:05.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fe5908a6640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] conn(0x7fe578070580 0x7fe5780729a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fe58b7fe640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe578068380 0x7fe57806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fe5908a6640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] conn(0x7fe578070580 0x7fe5780729a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fe588ff9640 1 -- 172.21.15.62:0/3025234477 <== osd.0 v2:172.21.15.62:6817/3805589275 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe57c016ec0 con 0x7fe578070580 2024-04-18T10:06:05.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f7cd01c6640 1 -- 172.21.15.62:0/2691087558 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7cc00c8670 con 0x7f7cc0034370 2024-04-18T10:06:05.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.966+0000 7f7cbe7fc640 1 -- 172.21.15.62:0/2691087558 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cc4005220 con 0x7f7cc0034370 2024-04-18T10:06:05.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f7cbe7fc640 1 -- 172.21.15.62:0/2691087558 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc4007030 con 0x7f7cc0034370 2024-04-18T10:06:05.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f88e8ff9640 1 -- 172.21.15.62:0/1143537152 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f88e40a7070 con 0x7f88ec0b1ed0 2024-04-18T10:06:05.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f88e8ff9640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f88d80683b0 0x7f88d806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f88e8ff9640 1 -- 172.21.15.62:0/1143537152 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f88e4120ff0 con 0x7f88ec0b1ed0 2024-04-18T10:06:05.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f88e8ff9640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] conn(0x7f88d80705b0 0x7f88d80729d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f88e8ff9640 1 -- 172.21.15.62:0/1143537152 --> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88e40b4e50 con 0x7f88d80705b0 2024-04-18T10:06:05.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f88e8ff9640 1 -- 172.21.15.62:0/1143537152 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f88e41213e0 con 0x7f88ec0b1ed0 2024-04-18T10:06:05.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f88f0ec7640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] conn(0x7f88d80705b0 0x7f88d80729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f88eb7fe640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f88d80683b0 0x7f88d806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f88f0ec7640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] conn(0x7f88d80705b0 0x7f88d80729d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f88eb7fe640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f88d80683b0 0x7f88d806a870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f88ec144b80 tx=0x7f88e00090e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fd62b7fe640 1 -- 172.21.15.62:0/50474965 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63800a460 con 0x7fd644073aa0 2024-04-18T10:06:05.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f7142ffd640 1 -- 172.21.15.62:0/2875069765 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7144011070 con 0x7f715414c410 2024-04-18T10:06:05.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f7142ffd640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7134068450 0x7f713406a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f7142ffd640 1 -- 172.21.15.62:0/2875069765 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7144086de0 con 0x7f715414c410 2024-04-18T10:06:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f7142ffd640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] conn(0x7f7134070670 0x7f7134072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f7142ffd640 1 -- 172.21.15.62:0/2875069765 --> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f714401ee50 con 0x7f7134070670 2024-04-18T10:06:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f7142ffd640 1 -- 172.21.15.62:0/2875069765 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f71440871d0 con 0x7f715414c410 2024-04-18T10:06:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f1890ff9640 1 -- 172.21.15.62:0/3650078111 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1888018020 con 0x7f18940b9e10 2024-04-18T10:06:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f1890ff9640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f18800683b0 0x7f188006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f1890ff9640 1 -- 172.21.15.62:0/3650078111 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1888086ea0 con 0x7f18940b9e10 2024-04-18T10:06:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f1890ff9640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] conn(0x7f1880070630 0x7f1880072a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f1890ff9640 1 -- 172.21.15.62:0/3650078111 --> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f188800de50 con 0x7f1880070630 2024-04-18T10:06:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f1890ff9640 1 -- 172.21.15.62:0/3650078111 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f1888087290 con 0x7f18940b9e10 2024-04-18T10:06:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f18937fe640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] conn(0x7f1880070630 0x7f1880072a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f1893fff640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f18800683b0 0x7f188006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fe58b7fe640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe578068380 0x7fe57806a840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe5840ae000 tx=0x7fe584004ac0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fe588ff9640 1 -- 172.21.15.62:0/3025234477 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe57c051e60 con 0x7fe58c0b2c60 2024-04-18T10:06:05.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7fb014ff9640 1 -- 172.21.15.62:0/2957968992 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb018067810 con 0x7fb01001f1a0 2024-04-18T10:06:05.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f7151d9b640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] conn(0x7f7134070670 0x7f7134072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f715259c640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7134068450 0x7f713406a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f715259c640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7134068450 0x7f713406a910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7154148380 tx=0x7f7148008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.970+0000 7f7151d9b640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] conn(0x7f7134070670 0x7f7134072a90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f18937fe640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] conn(0x7f1880070630 0x7f1880072a50 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f1893fff640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f18800683b0 0x7f188006a870 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f188c002080 tx=0x7f188c001670 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f06c0c74640 1 -- 172.21.15.62:0/1303004177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06bc147d50 con 0x7f06bc071f30 2024-04-18T10:06:05.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f06c0c74640 1 -- 172.21.15.62:0/1303004177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06bc1e5380 con 0x7f06bc071f30 2024-04-18T10:06:05.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f06c0c74640 1 -- 172.21.15.62:0/1303004177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f06bc04fe10 con 0x7f06bc071f30 2024-04-18T10:06:05.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fd62b7fe640 1 -- 172.21.15.62:0/50474965 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd63800a600 con 0x7fd644073aa0 2024-04-18T10:06:05.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fd62b7fe640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd630068380 0x7fd63006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fd62b7fe640 1 -- 172.21.15.62:0/50474965 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd638120d80 con 0x7fd644073aa0 2024-04-18T10:06:05.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fd62b7fe640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] conn(0x7fd630070600 0x7fd630072a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fd62b7fe640 1 -- 172.21.15.62:0/50474965 --> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6300730d0 con 0x7fd630070600 2024-04-18T10:06:05.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fd62b7fe640 1 -- 172.21.15.62:0/50474965 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fd638121170 con 0x7fd644073aa0 2024-04-18T10:06:05.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fb014ff9640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faffc068380 0x7faffc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f1890ff9640 1 -- 172.21.15.62:0/3650078111 <== osd.7 v2:172.21.15.145:6802/69473053 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f188800de50 con 0x7f1880070630 2024-04-18T10:06:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fd6427fc640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd630068380 0x7fd63006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fb014ff9640 1 -- 172.21.15.62:0/2957968992 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb018102760 con 0x7fb01001f1a0 2024-04-18T10:06:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fb017fff640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faffc068380 0x7faffc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fd6437fe640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] conn(0x7fd630070600 0x7fd630072a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f88e8ff9640 1 -- 172.21.15.62:0/1143537152 <== osd.2 v2:172.21.15.62:6816/1732263675 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f88e40b4e50 con 0x7f88d80705b0 2024-04-18T10:06:05.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fb014ff9640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] conn(0x7faffc070600 0x7faffc072a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fb014ff9640 1 -- 172.21.15.62:0/2957968992 --> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb0180758f0 con 0x7faffc070600 2024-04-18T10:06:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fb014ff9640 1 -- 172.21.15.62:0/2957968992 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fb018102b50 con 0x7fb01001f1a0 2024-04-18T10:06:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f06aaffd640 1 -- 172.21.15.62:0/1303004177 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f06ac0b5070 con 0x7f06bc071f30 2024-04-18T10:06:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f06aaffd640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f069c068380 0x7f069c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f06aaffd640 1 -- 172.21.15.62:0/1303004177 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f06ac11fd20 con 0x7f06bc071f30 2024-04-18T10:06:05.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f06aaffd640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] conn(0x7f069c070520 0x7f069c072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f06aaffd640 1 -- 172.21.15.62:0/1303004177 --> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06ac00af30 con 0x7f069c070520 2024-04-18T10:06:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fb01cd5d640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] conn(0x7faffc070600 0x7faffc072a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f06ba59c640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f069c068380 0x7f069c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fd6427fc640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd630068380 0x7fd63006a840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd6340011f0 tx=0x7fd634007d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f7cbe7fc640 1 -- 172.21.15.62:0/2691087558 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7cc40071d0 con 0x7f7cc0034370 2024-04-18T10:06:05.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f7cbe7fc640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7cac068450 0x7f7cac06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f7cbe7fc640 1 -- 172.21.15.62:0/2691087558 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7cc4087030 con 0x7f7cc0034370 2024-04-18T10:06:05.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f7cbe7fc640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] conn(0x7f7cac0706d0 0x7f7cac072af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f7cbe7fc640 1 -- 172.21.15.62:0/2691087558 --> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7cc4017e70 con 0x7f7cac0706d0 2024-04-18T10:06:05.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f7cbe7fc640 1 -- 172.21.15.62:0/2691087558 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7cc4087420 con 0x7f7cc0034370 2024-04-18T10:06:05.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f7ccd761640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7cac068450 0x7f7cac06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f7cce763640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] conn(0x7f7cac0706d0 0x7f7cac072af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.978+0000 7f7ccd761640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7cac068450 0x7f7cac06a910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80ad000 tx=0x7f7cb8004740 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7fb017fff640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faffc068380 0x7faffc06a840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0ad000 tx=0x7fb00c0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.974+0000 7f7142ffd640 1 -- 172.21.15.62:0/2875069765 <== osd.6 v2:172.21.15.145:6800/2070684678 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f714401ee50 con 0x7f7134070670 2024-04-18T10:06:05.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.978+0000 7fb01cd5d640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] conn(0x7faffc070600 0x7faffc072a20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.978+0000 7fd6437fe640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] conn(0x7fd630070600 0x7fd630072a20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.978+0000 7f06bad9d640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] conn(0x7f069c070520 0x7f069c072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.978+0000 7f7cce763640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] conn(0x7f7cac0706d0 0x7f7cac072af0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.978+0000 7f06ba59c640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f069c068380 0x7f069c06a840 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f06b000b240 tx=0x7f06b000b3a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.978+0000 7f06aaffd640 1 -- 172.21.15.62:0/1303004177 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f06ac0ebfa0 con 0x7f06bc071f30 2024-04-18T10:06:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.978+0000 7f06bad9d640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] conn(0x7f069c070520 0x7f069c072940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.978+0000 7fd62b7fe640 1 -- 172.21.15.62:0/50474965 <== osd.3 v2:172.21.15.62:6806/1675715975 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd6300730d0 con 0x7fd630070600 2024-04-18T10:06:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.978+0000 7f7cbe7fc640 1 -- 172.21.15.62:0/2691087558 <== osd.1 v2:172.21.15.62:6800/4072301756 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7cc4017e70 con 0x7f7cac0706d0 2024-04-18T10:06:05.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.978+0000 7fb014ff9640 1 -- 172.21.15.62:0/2957968992 <== osd.4 v2:172.21.15.145:6816/4248694380 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb0180758f0 con 0x7faffc070600 2024-04-18T10:06:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.986+0000 7f06aaffd640 1 -- 172.21.15.62:0/1303004177 <== osd.5 v2:172.21.15.145:6824/3413756159 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06ac00af30 con 0x7f069c070520 2024-04-18T10:06:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.994+0000 7f88f292a640 1 -- 172.21.15.62:0/1143537152 --> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f88ec0b3180 con 0x7f88d80705b0 2024-04-18T10:06:05.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.998+0000 7f88e8ff9640 1 -- 172.21.15.62:0/1143537152 <== osd.2 v2:172.21.15.62:6816/1732263675 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f88ec0b3180 con 0x7f88d80705b0 2024-04-18T10:06:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.998+0000 7f88d27fc640 1 -- 172.21.15.62:0/1143537152 >> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] conn(0x7f88d80705b0 msgr2=0x7f88d80729d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.998+0000 7f88d27fc640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] conn(0x7f88d80705b0 0x7f88d80729d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.998+0000 7f88d27fc640 1 -- 172.21.15.62:0/1143537152 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f88d80683b0 msgr2=0x7f88d806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.998+0000 7f88d27fc640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f88d80683b0 0x7f88d806a870 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f88ec144b80 tx=0x7f88e00090e0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.998+0000 7f88d27fc640 1 -- 172.21.15.62:0/1143537152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88ec0b1ed0 msgr2=0x7f88ec149500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.998+0000 7f88d27fc640 1 --2- 172.21.15.62:0/1143537152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f88ec0b1ed0 0x7f88ec149500 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f88e40aa000 tx=0x7f88e4019450 comp rx=0 tx=0).stop 2024-04-18T10:06:05.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.998+0000 7f88f0ec7640 1 -- 172.21.15.62:0/1143537152 reap_dead start 2024-04-18T10:06:05.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.998+0000 7f88d27fc640 1 -- 172.21.15.62:0/1143537152 shutdown_connections 2024-04-18T10:06:05.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.998+0000 7f88d27fc640 1 -- 172.21.15.62:0/1143537152 >> 172.21.15.62:0/1143537152 conn(0x7f88ec0b92f0 msgr2=0x7f88ec074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.998+0000 7f88d27fc640 1 -- 172.21.15.62:0/1143537152 shutdown_connections 2024-04-18T10:06:05.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:04.998+0000 7f88d27fc640 1 -- 172.21.15.62:0/1143537152 wait complete. 2024-04-18T10:06:05.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe592309640 1 -- 172.21.15.62:0/3025234477 --> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe56c002de0 con 0x7fe578070580 2024-04-18T10:06:05.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe588ff9640 1 -- 172.21.15.62:0/3025234477 <== osd.0 v2:172.21.15.62:6817/3805589275 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe56c002de0 con 0x7fe578070580 2024-04-18T10:06:05.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7f7cd01c6640 1 -- 172.21.15.62:0/2691087558 --> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7cc00053a0 con 0x7f7cac0706d0 2024-04-18T10:06:05.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe592309640 1 -- 172.21.15.62:0/3025234477 >> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] conn(0x7fe578070580 msgr2=0x7fe5780729a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe592309640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] conn(0x7fe578070580 0x7fe5780729a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe592309640 1 -- 172.21.15.62:0/3025234477 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe578068380 msgr2=0x7fe57806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe592309640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe578068380 0x7fe57806a840 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe5840ae000 tx=0x7fe584004ac0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe592309640 1 -- 172.21.15.62:0/3025234477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c0b2c60 msgr2=0x7fe58c142de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe592309640 1 --2- 172.21.15.62:0/3025234477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe58c0b2c60 0x7fe58c142de0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00c3f0 tx=0x7fe57c00c8b0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe5908a6640 1 -- 172.21.15.62:0/3025234477 reap_dead start 2024-04-18T10:06:05.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe592309640 1 -- 172.21.15.62:0/3025234477 shutdown_connections 2024-04-18T10:06:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe592309640 1 -- 172.21.15.62:0/3025234477 >> 172.21.15.62:0/3025234477 conn(0x7fe58c0b92f0 msgr2=0x7fe58c0b96d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe592309640 1 -- 172.21.15.62:0/3025234477 shutdown_connections 2024-04-18T10:06:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7fe592309640 1 -- 172.21.15.62:0/3025234477 wait complete. 2024-04-18T10:06:05.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7f7cbe7fc640 1 -- 172.21.15.62:0/2691087558 <== osd.1 v2:172.21.15.62:6800/4072301756 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7cc00053a0 con 0x7f7cac0706d0 2024-04-18T10:06:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7f7cabfff640 1 -- 172.21.15.62:0/2691087558 >> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] conn(0x7f7cac0706d0 msgr2=0x7f7cac072af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7f7cabfff640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] conn(0x7f7cac0706d0 0x7f7cac072af0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7f7cabfff640 1 -- 172.21.15.62:0/2691087558 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7cac068450 msgr2=0x7f7cac06a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7f7cabfff640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7cac068450 0x7f7cac06a910 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80ad000 tx=0x7f7cb8004740 comp rx=0 tx=0).stop 2024-04-18T10:06:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7f7cabfff640 1 -- 172.21.15.62:0/2691087558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cc0034370 msgr2=0x7f7cc00c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7f7cabfff640 1 --2- 172.21.15.62:0/2691087558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cc0034370 0x7f7cc00c8dc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7cc400c500 tx=0x7f7cc400c9c0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.002+0000 7f7cce763640 1 -- 172.21.15.62:0/2691087558 reap_dead start 2024-04-18T10:06:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.006+0000 7fd64952a640 1 -- 172.21.15.62:0/50474965 --> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd644148260 con 0x7fd630070600 2024-04-18T10:06:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.006+0000 7fd62b7fe640 1 -- 172.21.15.62:0/50474965 <== osd.3 v2:172.21.15.62:6806/1675715975 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd644148260 con 0x7fd630070600 2024-04-18T10:06:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.006+0000 7fd64952a640 1 -- 172.21.15.62:0/50474965 >> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] conn(0x7fd630070600 msgr2=0x7fd630072a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.006+0000 7fd64952a640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] conn(0x7fd630070600 0x7fd630072a20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7fd64952a640 1 -- 172.21.15.62:0/50474965 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd630068380 msgr2=0x7fd63006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7fd64952a640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd630068380 0x7fd63006a840 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd6340011f0 tx=0x7fd634007d40 comp rx=0 tx=0).stop 2024-04-18T10:06:05.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7f7cabfff640 1 -- 172.21.15.62:0/2691087558 shutdown_connections 2024-04-18T10:06:05.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7f7cabfff640 1 -- 172.21.15.62:0/2691087558 >> 172.21.15.62:0/2691087558 conn(0x7f7cc003abd0 msgr2=0x7f7cc00055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7fd64952a640 1 -- 172.21.15.62:0/50474965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd644073aa0 msgr2=0x7fd6441e4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7f7cabfff640 1 -- 172.21.15.62:0/2691087558 shutdown_connections 2024-04-18T10:06:05.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7fd64952a640 1 --2- 172.21.15.62:0/50474965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd644073aa0 0x7fd6441e4440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd63801cca0 tx=0x7fd63801b390 comp rx=0 tx=0).stop 2024-04-18T10:06:05.066 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7f7cabfff640 1 -- 172.21.15.62:0/2691087558 wait complete. 2024-04-18T10:06:05.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7fd6437fe640 1 -- 172.21.15.62:0/50474965 reap_dead start 2024-04-18T10:06:05.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7fd64952a640 1 -- 172.21.15.62:0/50474965 shutdown_connections 2024-04-18T10:06:05.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7fd64952a640 1 -- 172.21.15.62:0/50474965 >> 172.21.15.62:0/50474965 conn(0x7fd6440b92f0 msgr2=0x7fd6440b8320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7fd64952a640 1 -- 172.21.15.62:0/50474965 shutdown_connections 2024-04-18T10:06:05.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7fd64952a640 1 -- 172.21.15.62:0/50474965 wait complete. 2024-04-18T10:06:05.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7fb01d55e640 1 -- 172.21.15.62:0/2957968992 --> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faff8002d60 con 0x7faffc070600 2024-04-18T10:06:05.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7f189aa0a640 1 -- 172.21.15.62:0/3650078111 --> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1894133450 con 0x7f1880070630 2024-04-18T10:06:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7f715359e640 1 -- 172.21.15.62:0/2875069765 --> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7154156580 con 0x7f7134070670 2024-04-18T10:06:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7fb014ff9640 1 -- 172.21.15.62:0/2957968992 <== osd.4 v2:172.21.15.145:6816/4248694380 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faff8002d60 con 0x7faffc070600 2024-04-18T10:06:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7f1890ff9640 1 -- 172.21.15.62:0/3650078111 <== osd.7 v2:172.21.15.145:6802/69473053 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1894133450 con 0x7f1880070630 2024-04-18T10:06:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.010+0000 7f7142ffd640 1 -- 172.21.15.62:0/2875069765 <== osd.6 v2:172.21.15.145:6800/2070684678 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7154156580 con 0x7f7134070670 2024-04-18T10:06:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f7140ff9640 1 -- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] conn(0x7f7134070670 msgr2=0x7f7134072a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f7140ff9640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] conn(0x7f7134070670 0x7f7134072a90 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f189aa0a640 1 -- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] conn(0x7f1880070630 msgr2=0x7f1880072a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f189aa0a640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] conn(0x7f1880070630 0x7f1880072a50 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7fb01d55e640 1 -- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] conn(0x7faffc070600 msgr2=0x7faffc072a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7fb01d55e640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] conn(0x7faffc070600 0x7faffc072a20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f7140ff9640 1 -- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7134068450 msgr2=0x7f713406a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f7140ff9640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7134068450 0x7f713406a910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7154148380 tx=0x7f7148008040 comp rx=0 tx=0).stop 2024-04-18T10:06:05.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f7140ff9640 1 -- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f715414c410 msgr2=0x7f7154146e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f7140ff9640 1 --2- 172.21.15.62:0/2875069765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f715414c410 0x7f7154146e90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f714400c390 tx=0x7f714400c850 comp rx=0 tx=0).stop 2024-04-18T10:06:05.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f7152d9d640 1 -- 172.21.15.62:0/2875069765 reap_dead start 2024-04-18T10:06:05.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f7140ff9640 1 -- 172.21.15.62:0/2875069765 shutdown_connections 2024-04-18T10:06:05.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f7140ff9640 1 -- 172.21.15.62:0/2875069765 >> 172.21.15.62:0/2875069765 conn(0x7f71540b91f0 msgr2=0x7f71540b5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f7140ff9640 1 -- 172.21.15.62:0/2875069765 shutdown_connections 2024-04-18T10:06:05.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f7140ff9640 1 -- 172.21.15.62:0/2875069765 wait complete. 2024-04-18T10:06:05.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f06c0c74640 1 -- 172.21.15.62:0/1303004177 --> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f06bc0739d0 con 0x7f069c070520 2024-04-18T10:06:05.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f189aa0a640 1 -- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f18800683b0 msgr2=0x7f188006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f189aa0a640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f18800683b0 0x7f188006a870 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f188c002080 tx=0x7f188c001670 comp rx=0 tx=0).stop 2024-04-18T10:06:05.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f189aa0a640 1 -- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18940b9e10 msgr2=0x7f18941340e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f189aa0a640 1 --2- 172.21.15.62:0/3650078111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18940b9e10 0x7f18941340e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f188800b410 tx=0x7f188800b8d0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f1898fa7640 1 -- 172.21.15.62:0/3650078111 reap_dead start 2024-04-18T10:06:05.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7fb01d55e640 1 -- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faffc068380 msgr2=0x7faffc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7fb01d55e640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faffc068380 0x7faffc06a840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0ad000 tx=0x7fb00c0a6000 comp rx=0 tx=0).stop 2024-04-18T10:06:05.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7fb01d55e640 1 -- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb01001f1a0 msgr2=0x7fb010172110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7fb01d55e640 1 --2- 172.21.15.62:0/2957968992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb01001f1a0 0x7fb010172110 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb01806a910 tx=0x7fb018069600 comp rx=0 tx=0).stop 2024-04-18T10:06:05.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7fb01cd5d640 1 -- 172.21.15.62:0/2957968992 reap_dead start 2024-04-18T10:06:05.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7fb01d55e640 1 -- 172.21.15.62:0/2957968992 shutdown_connections 2024-04-18T10:06:05.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7fb01d55e640 1 -- 172.21.15.62:0/2957968992 >> 172.21.15.62:0/2957968992 conn(0x7fb01000b5e0 msgr2=0x7fb01003acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7fb01d55e640 1 -- 172.21.15.62:0/2957968992 shutdown_connections 2024-04-18T10:06:05.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7fb01d55e640 1 -- 172.21.15.62:0/2957968992 wait complete. 2024-04-18T10:06:05.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f189aa0a640 1 -- 172.21.15.62:0/3650078111 shutdown_connections 2024-04-18T10:06:05.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f189aa0a640 1 -- 172.21.15.62:0/3650078111 >> 172.21.15.62:0/3650078111 conn(0x7f18940b92f0 msgr2=0x7f18940b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f189aa0a640 1 -- 172.21.15.62:0/3650078111 shutdown_connections 2024-04-18T10:06:05.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f06aaffd640 1 -- 172.21.15.62:0/1303004177 <== osd.5 v2:172.21.15.145:6824/3413756159 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f06bc0739d0 con 0x7f069c070520 2024-04-18T10:06:05.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.014+0000 7f189aa0a640 1 -- 172.21.15.62:0/3650078111 wait complete. 2024-04-18T10:06:05.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.018+0000 7f06c0c74640 1 -- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] conn(0x7f069c070520 msgr2=0x7f069c072940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.018+0000 7f06c0c74640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] conn(0x7f069c070520 0x7f069c072940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.018+0000 7f06c0c74640 1 -- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f069c068380 msgr2=0x7f069c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.018+0000 7f06c0c74640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f069c068380 0x7f069c06a840 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f06b000b240 tx=0x7f06b000b3a0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.018+0000 7f06c0c74640 1 -- 172.21.15.62:0/1303004177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06bc071f30 msgr2=0x7f06bc13fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.018+0000 7f06c0c74640 1 --2- 172.21.15.62:0/1303004177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f06bc071f30 0x7f06bc13fd00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f06bc0b44a0 tx=0x7f06ac0191f0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.018+0000 7f06bad9d640 1 -- 172.21.15.62:0/1303004177 reap_dead start 2024-04-18T10:06:05.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.022+0000 7f06c0c74640 1 -- 172.21.15.62:0/1303004177 shutdown_connections 2024-04-18T10:06:05.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.022+0000 7f06c0c74640 1 -- 172.21.15.62:0/1303004177 >> 172.21.15.62:0/1303004177 conn(0x7f06bc0ba270 msgr2=0x7f06bc0b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.022+0000 7f06c0c74640 1 -- 172.21.15.62:0/1303004177 shutdown_connections 2024-04-18T10:06:05.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.022+0000 7f06c0c74640 1 -- 172.21.15.62:0/1303004177 wait complete. 2024-04-18T10:06:05.078 INFO:teuthology.orchestra.run.smithi062.stdout:55834574855 2024-04-18T10:06:05.079 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-18T10:06:05.080 INFO:teuthology.orchestra.run.smithi062.stdout:55834574854 2024-04-18T10:06:05.080 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-18T10:06:05.081 INFO:teuthology.orchestra.run.smithi062.stdout:55834574856 2024-04-18T10:06:05.081 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-18T10:06:05.082 INFO:teuthology.orchestra.run.smithi062.stdout:55834574855 2024-04-18T10:06:05.082 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-18T10:06:05.083 INFO:teuthology.orchestra.run.smithi062.stdout:55834574856 2024-04-18T10:06:05.084 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-18T10:06:05.084 INFO:teuthology.orchestra.run.smithi062.stdout:55834574856 2024-04-18T10:06:05.085 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-18T10:06:05.086 INFO:teuthology.orchestra.run.smithi062.stdout:55834574854 2024-04-18T10:06:05.086 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-18T10:06:05.087 INFO:teuthology.orchestra.run.smithi062.stdout:55834574856 2024-04-18T10:06:05.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-18T10:06:05.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84ffef640 1 Processor -- start 2024-04-18T10:06:05.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f38fd86d640 1 Processor -- start 2024-04-18T10:06:05.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84ffef640 1 -- start start 2024-04-18T10:06:05.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f38fd86d640 1 -- start start 2024-04-18T10:06:05.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f627ffff640 1 Processor -- start 2024-04-18T10:06:05.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f627ffff640 1 -- start start 2024-04-18T10:06:05.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f3cdaaed640 1 Processor -- start 2024-04-18T10:06:05.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84ffef640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8480b2370 0x7fd8480b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84ffef640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd848073aa0 0x7fd8480b5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84ffef640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8480b0a40 con 0x7fd8480b1880 2024-04-18T10:06:05.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84ffef640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fd8480b0d20 con 0x7fd848073aa0 2024-04-18T10:06:05.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84ffef640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd8480b1000 con 0x7fd8480b2370 2024-04-18T10:06:05.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f3cdaaed640 1 -- start start 2024-04-18T10:06:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f38fd86d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38f814c520 0x7f38f814c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f38fd86d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f38f81458a0 0x7f38f8145d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f38fd86d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38f80b1040 con 0x7f38f80b0c60 2024-04-18T10:06:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f38fd86d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f38f80b3630 con 0x7f38f81458a0 2024-04-18T10:06:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f38fd86d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f38f80b22f0 con 0x7f38f814c520 2024-04-18T10:06:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84e58c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd8480b1880 0x7fd848147100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54932/0 (socket says 172.21.15.62:54932) 2024-04-18T10:06:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84e58c640 1 -- 172.21.15.62:0/2871140182 learned_addr learned my addr 172.21.15.62:0/2871140182 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84d58a640 1 --2- 172.21.15.62:0/2871140182 >> v2:172.21.15.62:3300/0 conn(0x7fd848073aa0 0x7fd8480b5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84dd8b640 1 --2- 172.21.15.62:0/2871140182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8480b2370 0x7fd8480b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f627ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62801443f0 0x7f62800b2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f627ffff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f62800b2ca0 0x7f628014e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f627ffff640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f628013e5c0 con 0x7f628013e1e0 2024-04-18T10:06:05.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f627ffff640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f62800b22d0 con 0x7f62800b2ca0 2024-04-18T10:06:05.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f627ffff640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f6280144d40 con 0x7f62801443f0 2024-04-18T10:06:05.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f38f77fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f38f81458a0 0x7f38f8145d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f38f77fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f38f81458a0 0x7f38f8145d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53284/0 (socket says 172.21.15.62:53284) 2024-04-18T10:06:05.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f38f77fe640 1 -- 172.21.15.62:0/417545102 learned_addr learned my addr 172.21.15.62:0/417545102 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f627effd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f628013e1e0 0x7f62800b4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54934/0 (socket says 172.21.15.62:54934) 2024-04-18T10:06:05.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f627effd640 1 -- 172.21.15.62:0/1017120132 learned_addr learned my addr 172.21.15.62:0/1017120132 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:05.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f38f67fc640 1 --2- 172.21.15.62:0/417545102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38f814c520 0x7f38f814c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f0cc966c640 1 Processor -- start 2024-04-18T10:06:05.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f627f7fe640 1 --2- 172.21.15.62:0/1017120132 >> v2:172.21.15.145:3300/0 conn(0x7f62800b2ca0 0x7f628014e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f3cdaaed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cd406bd30 0x7f3cd40738c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f3cdaaed640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3cd4073e00 0x7f3cd4145b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f3cdaaed640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cd414f450 con 0x7f3cd40b69c0 2024-04-18T10:06:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f3cdaaed640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f3cd4150410 con 0x7f3cd4073e00 2024-04-18T10:06:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f3cdaaed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3cd4071720 con 0x7f3cd406bd30 2024-04-18T10:06:05.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f0cc966c640 1 -- start start 2024-04-18T10:06:05.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f38f77fe640 1 -- 172.21.15.62:0/417545102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38f814c520 msgr2=0x7f38f814c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:05.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f38f77fe640 1 --2- 172.21.15.62:0/417545102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38f814c520 0x7f38f814c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f38f77fe640 1 -- 172.21.15.62:0/417545102 >> v1:172.21.15.62:6789/0 conn(0x7f38f80b0c60 legacy=0x7f38f80b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:05.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f38f77fe640 1 -- 172.21.15.62:0/417545102 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f814c000 con 0x7f38f81458a0 2024-04-18T10:06:05.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f38f67fc640 1 --2- 172.21.15.62:0/417545102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38f814c520 0x7f38f814c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:05.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7f627e7fc640 1 --2- 172.21.15.62:0/1017120132 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62801443f0 0x7f62800b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd92ea640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3cd4073e00 0x7f3cd4145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f627f7fe640 1 -- 172.21.15.62:0/1017120132 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62801443f0 msgr2=0x7f62800b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:05.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f627f7fe640 1 --2- 172.21.15.62:0/1017120132 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62801443f0 0x7f62800b2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f627f7fe640 1 -- 172.21.15.62:0/1017120132 >> v1:172.21.15.62:6789/0 conn(0x7f628013e1e0 legacy=0x7f62800b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f627f7fe640 1 -- 172.21.15.62:0/1017120132 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62800b1f60 con 0x7f62800b2ca0 2024-04-18T10:06:05.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd92ea640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3cd4073e00 0x7f3cd4145b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53300/0 (socket says 172.21.15.62:53300) 2024-04-18T10:06:05.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd92ea640 1 -- 172.21.15.62:0/4075902177 learned_addr learned my addr 172.21.15.62:0/4075902177 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 225449485 0 0) 0x7f3cd414f450 con 0x7f3cd40b69c0 2024-04-18T10:06:05.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd9aeb640 1 --2- 172.21.15.62:0/4075902177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cd406bd30 0x7f3cd40738c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cc0003530 con 0x7f3cd40b69c0 2024-04-18T10:06:05.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc966c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146270 0x7f0cc414c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc966c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0cc40b22a0 0x7f0cc414c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc966c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cc40b0dc0 con 0x7f0cc40b1ec0 2024-04-18T10:06:05.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc966c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f0cc414bf30 con 0x7f0cc40b22a0 2024-04-18T10:06:05.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc966c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f0cc40b1060 con 0x7f0cc4146270 2024-04-18T10:06:05.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1920809145 0 0) 0x7f3cc0003530 con 0x7f3cd40b69c0 2024-04-18T10:06:05.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cd406bd30 msgr2=0x7f3cd40738c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:05.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 --2- 172.21.15.62:0/4075902177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cd406bd30 0x7f3cd40738c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 >> v2:172.21.15.62:3300/0 conn(0x7f3cd4073e00 msgr2=0x7f3cd4145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:05.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 --2- 172.21.15.62:0/4075902177 >> v2:172.21.15.62:3300/0 conn(0x7f3cd4073e00 0x7f3cd4145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd406c110 con 0x7f3cd40b69c0 2024-04-18T10:06:05.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f3cc40016a0 con 0x7f3cd40b69c0 2024-04-18T10:06:05.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 >> v1:172.21.15.62:6789/0 conn(0x7f3cd40b69c0 legacy=0x7f3cd40b4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc37fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146270 0x7f0cc414c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc8e6b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0cc40b22a0 0x7f0cc414c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc37fe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146270 0x7f0cc414c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:47882/0 (socket says 172.21.15.62:47882) 2024-04-18T10:06:05.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc37fe640 1 -- 172.21.15.62:0/1184873698 learned_addr learned my addr 172.21.15.62:0/1184873698 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84dd8b640 1 -- 172.21.15.62:0/2871140182 >> v2:172.21.15.62:3300/0 conn(0x7fd848073aa0 msgr2=0x7fd8480b5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:05.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84dd8b640 1 --2- 172.21.15.62:0/2871140182 >> v2:172.21.15.62:3300/0 conn(0x7fd848073aa0 0x7fd8480b5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.298+0000 7fd84dd8b640 1 -- 172.21.15.62:0/2871140182 >> v1:172.21.15.62:6789/0 conn(0x7fd8480b1880 legacy=0x7fd848147100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fd84dd8b640 1 -- 172.21.15.62:0/2871140182 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8480b2750 con 0x7fd8480b2370 2024-04-18T10:06:05.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 --2- 172.21.15.62:0/4075902177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cc00039c0 0x7f3cc0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 --2- 172.21.15.62:0/4075902177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cc0004810 0x7f3cc0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 --2- 172.21.15.62:0/4075902177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cd406bd30 0x7f3cc00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd4071720 con 0x7f3cc0004810 2024-04-18T10:06:05.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc000b9d0 con 0x7f3cc00039c0 2024-04-18T10:06:05.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3cc000bb50 con 0x7f3cd406bd30 2024-04-18T10:06:05.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fa5e3555640 1 Processor -- start 2024-04-18T10:06:05.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cda2ec640 1 --2- 172.21.15.62:0/4075902177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cd406bd30 0x7f3cc00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd9aeb640 1 --2- 172.21.15.62:0/4075902177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cc00039c0 0x7f3cc0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fa5e3555640 1 -- start start 2024-04-18T10:06:05.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc37fe640 1 -- 172.21.15.62:0/1184873698 >> v2:172.21.15.145:3300/0 conn(0x7f0cc40b22a0 msgr2=0x7f0cc414c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:05.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc37fe640 1 --2- 172.21.15.62:0/1184873698 >> v2:172.21.15.145:3300/0 conn(0x7f0cc40b22a0 0x7f0cc414c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc37fe640 1 -- 172.21.15.62:0/1184873698 >> v1:172.21.15.145:6789/0 conn(0x7f0cc40b1ec0 legacy=0x7f0cc4145b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:05.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f0cc37fe640 1 -- 172.21.15.62:0/1184873698 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cc40b1340 con 0x7f0cc4146270 2024-04-18T10:06:05.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd92ea640 1 --2- 172.21.15.62:0/4075902177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cc0004810 0x7f3cc0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f38f77fe640 1 --2- 172.21.15.62:0/417545102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38f81458a0 0x7f38f8145d10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009e50 tx=0x7f38f001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299b32dcf1e478dc server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd92ea640 1 -- 172.21.15.62:0/4075902177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cd406bd30 msgr2=0x7f3cc00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:05.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd92ea640 1 --2- 172.21.15.62:0/4075902177 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cd406bd30 0x7f3cc00051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd92ea640 1 -- 172.21.15.62:0/4075902177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cc00039c0 msgr2=0x7f3cc0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:05.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd92ea640 1 --2- 172.21.15.62:0/4075902177 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cc00039c0 0x7f3cc0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f3cd92ea640 1 -- 172.21.15.62:0/4075902177 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc0003530 con 0x7f3cc0004810 2024-04-18T10:06:05.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fa5e3555640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa5dc0b4aa0 0x7fa5dc149420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fa5e3555640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5dc098da0 con 0x7fa5dc0b8dc0 2024-04-18T10:06:05.314 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fa5e3555640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5dc0b9250 con 0x7fa5dc09c230 2024-04-18T10:06:05.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fa5e3555640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fa5dc0b53d0 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fd84dd8b640 1 --2- 172.21.15.62:0/2871140182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8480b2370 0x7fd8480b50c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0097c0 tx=0x7fd83c01e640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70c33c4d804cd2d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fa5e12f1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa5dc0b4aa0 0x7fa5dc149420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fa5e12f1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa5dc0b4aa0 0x7fa5dc149420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53306/0 (socket says 172.21.15.62:53306) 2024-04-18T10:06:05.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fa5e12f1640 1 -- 172.21.15.62:0/1945445340 learned_addr learned my addr 172.21.15.62:0/1945445340 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f2b81165640 1 Processor -- start 2024-04-18T10:06:05.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f2b81165640 1 -- start start 2024-04-18T10:06:05.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f2b81165640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2b74035940 0x7f2b74036540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f2b81165640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b7402a020 con 0x7f2b74035560 2024-04-18T10:06:05.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f2b81165640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b74033f10 con 0x7f2b740c8600 2024-04-18T10:06:05.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f2b81165640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f2b74037430 con 0x7f2b74035940 2024-04-18T10:06:05.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f2b7ad9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2b74035560 0x7f2b74033800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54960/0 (socket says 172.21.15.62:54960) 2024-04-18T10:06:05.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f2b7ad9d640 1 -- 172.21.15.62:0/2603986304 learned_addr learned my addr 172.21.15.62:0/2603986304 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:05.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f38f5ffb640 1 -- 172.21.15.62:0/417545102 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f00a5770 con 0x7f38f81458a0 2024-04-18T10:06:05.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f2b79d9b640 1 -- 172.21.15.62:0/2603986304 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3942508464 0 0) 0x7f2b7402a020 con 0x7f2b74035560 2024-04-18T10:06:05.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f2b79d9b640 1 -- 172.21.15.62:0/2603986304 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b64003530 con 0x7f2b74035560 2024-04-18T10:06:05.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f2b7a59c640 1 --2- 172.21.15.62:0/2603986304 >> v2:172.21.15.62:3300/0 conn(0x7f2b74035940 0x7f2b74036540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7f38f5ffb640 1 -- 172.21.15.62:0/417545102 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38f00a5910 con 0x7f38f81458a0 2024-04-18T10:06:05.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f627f7fe640 1 --2- 172.21.15.62:0/1017120132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62800b2ca0 0x7f628014e840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f62740097c0 tx=0x7f627401e6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a649c5afd180f9e server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fa5e12f1640 1 -- 172.21.15.62:0/1945445340 >> v1:172.21.15.145:6789/0 conn(0x7fa5dc09c230 legacy=0x7fa5dc0b7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.302+0000 7fa5e12f1640 1 -- 172.21.15.62:0/1945445340 >> v1:172.21.15.62:6789/0 conn(0x7fa5dc0b8dc0 legacy=0x7fa5dc1569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fa5e12f1640 1 -- 172.21.15.62:0/1945445340 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5dc0b81f0 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fd84cd89640 1 -- 172.21.15.62:0/2871140182 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0ae070 con 0x7fd8480b2370 2024-04-18T10:06:05.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fd84cd89640 1 -- 172.21.15.62:0/2871140182 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd83c0a8610 con 0x7fd8480b2370 2024-04-18T10:06:05.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fd84cd89640 1 -- 172.21.15.62:0/2871140182 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0a89e0 con 0x7fd8480b2370 2024-04-18T10:06:05.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38f5ffb640 1 -- 172.21.15.62:0/417545102 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f00a5ce0 con 0x7f38f81458a0 2024-04-18T10:06:05.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f627dffb640 1 -- 172.21.15.62:0/1017120132 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62740ad640 con 0x7f62800b2ca0 2024-04-18T10:06:05.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f627dffb640 1 -- 172.21.15.62:0/1017120132 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62740b2070 con 0x7f62800b2ca0 2024-04-18T10:06:05.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f2b7a59c640 1 -- 172.21.15.62:0/2603986304 >> v1:172.21.15.145:6789/0 conn(0x7f2b740c8600 legacy=0x7f2b740d2ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-18T10:06:05.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f2b7a59c640 1 -- 172.21.15.62:0/2603986304 >> v1:172.21.15.62:6789/0 conn(0x7f2b74035560 legacy=0x7f2b74033800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f2b7a59c640 1 -- 172.21.15.62:0/2603986304 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b74034e10 con 0x7f2b74035940 2024-04-18T10:06:05.320 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f627dffb640 1 -- 172.21.15.62:0/1017120132 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62740ada00 con 0x7f62800b2ca0 2024-04-18T10:06:05.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 -- 172.21.15.62:0/417545102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38f81458a0 msgr2=0x7f38f8145d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 --2- 172.21.15.62:0/417545102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38f81458a0 0x7f38f8145d10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009e50 tx=0x7f38f001c9a0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 -- 172.21.15.62:0/417545102 shutdown_connections 2024-04-18T10:06:05.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 --2- 172.21.15.62:0/417545102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38f81458a0 0x7f38f8145d10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 --2- 172.21.15.62:0/417545102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38f814c520 0x7f38f814c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 -- 172.21.15.62:0/417545102 >> 172.21.15.62:0/417545102 conn(0x7f38f80b92f0 msgr2=0x7f38f80b9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 -- 172.21.15.62:0/417545102 shutdown_connections 2024-04-18T10:06:05.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 -- 172.21.15.62:0/417545102 wait complete. 2024-04-18T10:06:05.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f627ffff640 1 -- 172.21.15.62:0/1017120132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62800b2ca0 msgr2=0x7f628014e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f0cc37fe640 1 --2- 172.21.15.62:0/1184873698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146270 0x7f0cc414c410 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0cc40b0f90 tx=0x7f0cb801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=549166ad5993d3ff server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f627ffff640 1 --2- 172.21.15.62:0/1017120132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62800b2ca0 0x7f628014e840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f62740097c0 tx=0x7f627401e6b0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f3cd92ea640 1 --2- 172.21.15.62:0/4075902177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cc0004810 0x7f3cc0004c90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00e1f0 tx=0x7f3ccc01cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1a4ed269f14359 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f627ffff640 1 -- 172.21.15.62:0/1017120132 shutdown_connections 2024-04-18T10:06:05.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f627ffff640 1 --2- 172.21.15.62:0/1017120132 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62800b2ca0 0x7f628014e840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f627ffff640 1 --2- 172.21.15.62:0/1017120132 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62801443f0 0x7f62800b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f627ffff640 1 -- 172.21.15.62:0/1017120132 >> 172.21.15.62:0/1017120132 conn(0x7f62800b91f0 msgr2=0x7f62800b9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f0cc2ffd640 1 -- 172.21.15.62:0/1184873698 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb80b0070 con 0x7f0cc4146270 2024-04-18T10:06:05.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f627ffff640 1 -- 172.21.15.62:0/1017120132 shutdown_connections 2024-04-18T10:06:05.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f627ffff640 1 -- 172.21.15.62:0/1017120132 wait complete. 2024-04-18T10:06:05.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 Processor -- start 2024-04-18T10:06:05.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc009cd0 con 0x7f3cc0004810 2024-04-18T10:06:05.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ccc0a5950 con 0x7f3cc0004810 2024-04-18T10:06:05.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f3cd8ae9640 1 -- 172.21.15.62:0/4075902177 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc0a5d20 con 0x7f3cc0004810 2024-04-18T10:06:05.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fa5e12f1640 1 --2- 172.21.15.62:0/1945445340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5dc0b4aa0 0x7fa5dc149420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc009870 tx=0x7fa5cc01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdef3d0fd36aad3c server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fa5d3fff640 1 -- 172.21.15.62:0/1945445340 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc0b0070 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fa5d3fff640 1 -- 172.21.15.62:0/1945445340 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5cc0ab400 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fa5d3fff640 1 -- 172.21.15.62:0/1945445340 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc0ab7d0 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fd84ffef640 1 -- 172.21.15.62:0/2871140182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8480b2370 msgr2=0x7fd8480b50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fd84ffef640 1 --2- 172.21.15.62:0/2871140182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8480b2370 0x7fd8480b50c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0097c0 tx=0x7fd83c01e640 comp rx=0 tx=0).stop 2024-04-18T10:06:05.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fd84ffef640 1 -- 172.21.15.62:0/2871140182 shutdown_connections 2024-04-18T10:06:05.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fd84ffef640 1 --2- 172.21.15.62:0/2871140182 >> v2:172.21.15.62:3300/0 conn(0x7fd848073aa0 0x7fd8480b5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fd84ffef640 1 --2- 172.21.15.62:0/2871140182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8480b2370 0x7fd8480b50c0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd83c0097c0 tx=0x7fd83c01e640 comp rx=0 tx=0).stop 2024-04-18T10:06:05.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fd84ffef640 1 -- 172.21.15.62:0/2871140182 >> 172.21.15.62:0/2871140182 conn(0x7fd8480b92f0 msgr2=0x7fd8480b9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f0cc2ffd640 1 -- 172.21.15.62:0/1184873698 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cb80ab600 con 0x7f0cc4146270 2024-04-18T10:06:05.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f0cc2ffd640 1 -- 172.21.15.62:0/1184873698 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb80ab9d0 con 0x7f0cc4146270 2024-04-18T10:06:05.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f0cc966c640 1 -- 172.21.15.62:0/1184873698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146270 msgr2=0x7f0cc414c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f0cc966c640 1 --2- 172.21.15.62:0/1184873698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146270 0x7f0cc414c410 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0cc40b0f90 tx=0x7f0cb801c470 comp rx=0 tx=0).stop 2024-04-18T10:06:05.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 -- start start 2024-04-18T10:06:05.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f0cc966c640 1 -- 172.21.15.62:0/1184873698 shutdown_connections 2024-04-18T10:06:05.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f0cc966c640 1 --2- 172.21.15.62:0/1184873698 >> v2:172.21.15.145:3300/0 conn(0x7f0cc40b22a0 0x7f0cc414c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f0cc966c640 1 --2- 172.21.15.62:0/1184873698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146270 0x7f0cc414c410 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0cc40b0f90 tx=0x7f0cb801c470 comp rx=0 tx=0).stop 2024-04-18T10:06:05.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f0cc966c640 1 -- 172.21.15.62:0/1184873698 >> 172.21.15.62:0/1184873698 conn(0x7f0cc40b91f0 msgr2=0x7f0cc40b9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fd84ffef640 1 -- 172.21.15.62:0/2871140182 shutdown_connections 2024-04-18T10:06:05.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f2b7a59c640 1 --2- 172.21.15.62:0/2603986304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b74035940 0x7f2b74036540 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2b700097c0 tx=0x7f2b7001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87f35a5b5397064c server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38f80b0c60 0x7f38f81e4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38f81458a0 0x7f38f813da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38f814c520 0x7f38f81e4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8159130 con 0x7f38f80b0c60 2024-04-18T10:06:05.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f38f8158fb0 con 0x7f38f814c520 2024-04-18T10:06:05.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38fd86d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f38f81592b0 con 0x7f38f81458a0 2024-04-18T10:06:05.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38f6ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38f80b0c60 0x7f38f81e4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38f6ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38f80b0c60 0x7f38f81e4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53326/0 (socket says 172.21.15.62:53326) 2024-04-18T10:06:05.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f38f6ffd640 1 -- 172.21.15.62:0/1180345932 learned_addr learned my addr 172.21.15.62:0/1180345932 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f2b79d9b640 1 -- 172.21.15.62:0/2603986304 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b700b0070 con 0x7f2b74035940 2024-04-18T10:06:05.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38f67fc640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38f81458a0 0x7f38f813da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f2b79d9b640 1 -- 172.21.15.62:0/2603986304 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b700ab600 con 0x7f2b74035940 2024-04-18T10:06:05.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f2b79d9b640 1 -- 172.21.15.62:0/2603986304 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b700a4e60 con 0x7f2b74035940 2024-04-18T10:06:05.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 -- 172.21.15.62:0/4075902177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cc0004810 msgr2=0x7f3cc0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 --2- 172.21.15.62:0/4075902177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cc0004810 0x7f3cc0004c90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00e1f0 tx=0x7f3ccc01cd30 comp rx=0 tx=0).stop 2024-04-18T10:06:05.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cda2ec640 1 -- 172.21.15.62:0/4075902177 reap_dead start 2024-04-18T10:06:05.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 -- 172.21.15.62:0/4075902177 shutdown_connections 2024-04-18T10:06:05.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 -- 172.21.15.62:0/4075902177 >> 172.21.15.62:0/4075902177 conn(0x7f3cd40ba370 msgr2=0x7f3cd40ba780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 -- 172.21.15.62:0/4075902177 shutdown_connections 2024-04-18T10:06:05.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7fd84ffef640 1 -- 172.21.15.62:0/2871140182 wait complete. 2024-04-18T10:06:05.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38f6ffd640 1 -- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38f81458a0 msgr2=0x7f38f813da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38f6ffd640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38f81458a0 0x7f38f813da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38f6ffd640 1 -- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38f814c520 msgr2=0x7f38f81e4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38f6ffd640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38f814c520 0x7f38f81e4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38f6ffd640 1 -- 172.21.15.62:0/1180345932 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38f8143bb0 con 0x7f38f80b0c60 2024-04-18T10:06:05.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38f67fc640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38f81458a0 0x7f38f813da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:06:05.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fa5e3555640 1 -- 172.21.15.62:0/1945445340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5dc0b4aa0 msgr2=0x7fa5dc149420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fa5e3555640 1 --2- 172.21.15.62:0/1945445340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5dc0b4aa0 0x7fa5dc149420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc009870 tx=0x7fa5cc01c4d0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fa5e3555640 1 -- 172.21.15.62:0/1945445340 shutdown_connections 2024-04-18T10:06:05.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fa5e3555640 1 --2- 172.21.15.62:0/1945445340 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5dc0b4aa0 0x7fa5dc149420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fa5e3555640 1 -- 172.21.15.62:0/1945445340 >> 172.21.15.62:0/1945445340 conn(0x7fa5dc09b530 msgr2=0x7fa5dc09b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f0cc966c640 1 -- 172.21.15.62:0/1184873698 shutdown_connections 2024-04-18T10:06:05.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.306+0000 7f0cc966c640 1 -- 172.21.15.62:0/1184873698 wait complete. 2024-04-18T10:06:05.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84ffef640 1 Processor -- start 2024-04-18T10:06:05.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f2b81165640 1 -- 172.21.15.62:0/2603986304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b74035940 msgr2=0x7f2b74036540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f2b81165640 1 --2- 172.21.15.62:0/2603986304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b74035940 0x7f2b74036540 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2b700097c0 tx=0x7f2b7001c3d0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f2b81165640 1 -- 172.21.15.62:0/2603986304 shutdown_connections 2024-04-18T10:06:05.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f2b81165640 1 --2- 172.21.15.62:0/2603986304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b74035940 0x7f2b74036540 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2b700097c0 tx=0x7f2b7001c3d0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f2b81165640 1 -- 172.21.15.62:0/2603986304 >> 172.21.15.62:0/2603986304 conn(0x7f2b7403abd0 msgr2=0x7f2b74005ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 -- 172.21.15.62:0/4075902177 wait complete. 2024-04-18T10:06:05.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38f6ffd640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38f80b0c60 0x7f38f81e4230 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f38e800c500 tx=0x7f38e800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f2b81165640 1 -- 172.21.15.62:0/2603986304 shutdown_connections 2024-04-18T10:06:05.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc966c640 1 Processor -- start 2024-04-18T10:06:05.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f2b81165640 1 -- 172.21.15.62:0/2603986304 wait complete. 2024-04-18T10:06:05.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fa5e3555640 1 -- 172.21.15.62:0/1945445340 shutdown_connections 2024-04-18T10:06:05.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 Processor -- start 2024-04-18T10:06:05.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84ffef640 1 -- start start 2024-04-18T10:06:05.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84ffef640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd848073aa0 0x7fd848141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84ffef640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8480b1880 0x7fd8481dfb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84ffef640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8481e0080 0x7fd8481d5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84ffef640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd848150f90 con 0x7fd848073aa0 2024-04-18T10:06:05.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84ffef640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fd848150e10 con 0x7fd8480b1880 2024-04-18T10:06:05.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84ffef640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd848151110 con 0x7fd8481e0080 2024-04-18T10:06:05.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84e58c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8481e0080 0x7fd8481d5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84dd8b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd848073aa0 0x7fd848141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84dd8b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd848073aa0 0x7fd848141bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53330/0 (socket says 172.21.15.62:53330) 2024-04-18T10:06:05.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84dd8b640 1 -- 172.21.15.62:0/4242478063 learned_addr learned my addr 172.21.15.62:0/4242478063 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f627ffff640 1 Processor -- start 2024-04-18T10:06:05.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f627ffff640 1 -- start start 2024-04-18T10:06:05.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc966c640 1 -- start start 2024-04-18T10:06:05.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc966c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cc40b1ec0 0x7f0cc413fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc966c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cc40b22a0 0x7f0cc4147ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc966c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146900 0x7f0cc413c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc966c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc4159350 con 0x7f0cc40b22a0 2024-04-18T10:06:05.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc966c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc41591d0 con 0x7f0cc40b1ec0 2024-04-18T10:06:05.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc966c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f0cc41594d0 con 0x7f0cc4146900 2024-04-18T10:06:05.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f2b81165640 1 Processor -- start 2024-04-18T10:06:05.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc3fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cc40b1ec0 0x7f0cc413fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc3fff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cc40b1ec0 0x7f0cc413fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.62:41994/0 (socket says 172.21.15.62:41994) 2024-04-18T10:06:05.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc3fff640 1 -- 172.21.15.62:0/2647919618 learned_addr learned my addr 172.21.15.62:0/2647919618 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc8e6b640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146900 0x7f0cc413c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f627ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f628013e1e0 0x7f62801df830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f627ffff640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62801443f0 0x7f62801dfd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84d58a640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8480b1880 0x7fd8481dfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f627ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62801e02b0 0x7f628012c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f627ffff640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6280151120 con 0x7f628013e1e0 2024-04-18T10:06:05.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f627ffff640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f6280150fa0 con 0x7f62801443f0 2024-04-18T10:06:05.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f627ffff640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f62801512a0 con 0x7f62801e02b0 2024-04-18T10:06:05.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fa5e3555640 1 -- 172.21.15.62:0/1945445340 wait complete. 2024-04-18T10:06:05.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38dffff640 1 -- 172.21.15.62:0/1180345932 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e8016020 con 0x7f38f80b0c60 2024-04-18T10:06:05.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38fd86d640 1 -- 172.21.15.62:0/1180345932 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38f81e78b0 con 0x7f38f80b0c60 2024-04-18T10:06:05.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38fd86d640 1 -- 172.21.15.62:0/1180345932 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38f81e7e70 con 0x7f38f80b0c60 2024-04-18T10:06:05.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fa5e3555640 1 Processor -- start 2024-04-18T10:06:05.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38dffff640 1 -- 172.21.15.62:0/1180345932 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38e8005220 con 0x7f38f80b0c60 2024-04-18T10:06:05.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f38dffff640 1 -- 172.21.15.62:0/1180345932 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38e8007030 con 0x7f38f80b0c60 2024-04-18T10:06:05.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f2b81165640 1 -- start start 2024-04-18T10:06:05.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84dd8b640 1 -- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8481e0080 msgr2=0x7fd8481d5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84dd8b640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8481e0080 0x7fd8481d5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84dd8b640 1 -- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8480b1880 msgr2=0x7fd8481dfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84dd8b640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8480b1880 0x7fd8481dfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84dd8b640 1 -- 172.21.15.62:0/4242478063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd848141140 con 0x7fd848073aa0 2024-04-18T10:06:05.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f627e7fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62801443f0 0x7f62801dfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f627e7fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62801443f0 0x7f62801dfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.62:42010/0 (socket says 172.21.15.62:42010) 2024-04-18T10:06:05.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f627e7fc640 1 -- 172.21.15.62:0/822914824 learned_addr learned my addr 172.21.15.62:0/822914824 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f0cc37fe640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cc40b22a0 0x7f0cc4147ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f627f7fe640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62801e02b0 0x7f628012c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 -- start start 2024-04-18T10:06:05.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4142150 0x7f3cd4142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cd4148a30 0x7f3cd41e7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cd4148e90 0x7f3cd41e84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd40b5f90 con 0x7f3cd4142150 2024-04-18T10:06:05.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f3cd4071f30 con 0x7f3cd4148a30 2024-04-18T10:06:05.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cdaaed640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3cd4146490 con 0x7f3cd4148e90 2024-04-18T10:06:05.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cd9aeb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4142150 0x7f3cd4142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cd9aeb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4142150 0x7f3cd4142530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53356/0 (socket says 172.21.15.62:53356) 2024-04-18T10:06:05.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cd9aeb640 1 -- 172.21.15.62:0/1283845604 learned_addr learned my addr 172.21.15.62:0/1283845604 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7fd84dd8b640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd848073aa0 0x7fd848141bd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd83c01ccf0 tx=0x7fd83c01b3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cd9aeb640 1 -- 172.21.15.62:0/1283845604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cd4148e90 msgr2=0x7f3cd41e84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cd9aeb640 1 --2- 172.21.15.62:0/1283845604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cd4148e90 0x7f3cd41e84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cd9aeb640 1 -- 172.21.15.62:0/1283845604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cd4148a30 msgr2=0x7f3cd41e7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cd9aeb640 1 --2- 172.21.15.62:0/1283845604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cd4148a30 0x7f3cd41e7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.310+0000 7f3cd9aeb640 1 -- 172.21.15.62:0/1283845604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cd413fe50 con 0x7f3cd4142150 2024-04-18T10:06:05.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f0cc8e6b640 1 -- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cc40b1ec0 msgr2=0x7f0cc413fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f0cc8e6b640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cc40b1ec0 0x7f0cc413fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f0cc8e6b640 1 -- 172.21.15.62:0/2647919618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cc40b22a0 msgr2=0x7f0cc4147ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f0cc8e6b640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cc40b22a0 0x7f0cc4147ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f0cc8e6b640 1 -- 172.21.15.62:0/2647919618 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cc4147740 con 0x7f0cc4146900 2024-04-18T10:06:05.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f627effd640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f628013e1e0 0x7f62801df830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b81165640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b74035560 0x7f2b740bb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b81165640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b740c8600 0x7f2b740c1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b81165640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b740c4520 0x7f2b740c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b81165640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2b74035db0 con 0x7f2b740c8600 2024-04-18T10:06:05.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b81165640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f2b740362f0 con 0x7f2b74035560 2024-04-18T10:06:05.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b81165640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2b740db400 con 0x7f2b740c4520 2024-04-18T10:06:05.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f38fd86d640 1 -- 172.21.15.62:0/1180345932 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38d80052a0 con 0x7f38f80b0c60 2024-04-18T10:06:05.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e3555640 1 -- start start 2024-04-18T10:06:05.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b7a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b740c8600 0x7f2b740c1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b7a59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b740c8600 0x7f2b740c1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53372/0 (socket says 172.21.15.62:53372) 2024-04-18T10:06:05.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b7a59c640 1 -- 172.21.15.62:0/291389020 learned_addr learned my addr 172.21.15.62:0/291389020 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f627effd640 1 -- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62801e02b0 msgr2=0x7f628012c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f627effd640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62801e02b0 0x7f628012c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f627effd640 1 -- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62801443f0 msgr2=0x7f62801dfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f627effd640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62801443f0 0x7f62801dfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f627effd640 1 -- 172.21.15.62:0/822914824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6280143440 con 0x7f628013e1e0 2024-04-18T10:06:05.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f0cc8e6b640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146900 0x7f0cc413c340 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc007d20 tx=0x7f0cbc00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b7ad9d640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b74035560 0x7f2b740bb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b7b59e640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b740c4520 0x7f2b740c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f3cd9aeb640 1 --2- 172.21.15.62:0/1283845604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4142150 0x7f3cd4142530 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0011610 tx=0x7f3cd0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f0cc0ff9640 1 -- 172.21.15.62:0/2647919618 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc00edd0 con 0x7f0cc4146900 2024-04-18T10:06:05.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5efa4640 1 Processor -- start 2024-04-18T10:06:05.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e3555640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5dc0b4aa0 0x7fa5dc0b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f0cc0ff9640 1 -- 172.21.15.62:0/2647919618 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cbc0039a0 con 0x7f0cc4146900 2024-04-18T10:06:05.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f0cc0ff9640 1 -- 172.21.15.62:0/2647919618 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc003d90 con 0x7f0cc4146900 2024-04-18T10:06:05.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f627effd640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f628013e1e0 0x7f62801df830 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f627000cb10 tx=0x7f627000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e3555640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5dc0b8dc0 0x7fa5dc1eddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6267fff640 1 -- 172.21.15.62:0/822914824 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6270017070 con 0x7f628013e1e0 2024-04-18T10:06:05.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6267fff640 1 -- 172.21.15.62:0/822914824 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f627000ac70 con 0x7f628013e1e0 2024-04-18T10:06:05.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6267fff640 1 -- 172.21.15.62:0/822914824 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6270012600 con 0x7f628013e1e0 2024-04-18T10:06:05.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e3555640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5dc0b3150 0x7fa5dc073220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e3555640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa5dc15aca0 con 0x7fa5dc0b8dc0 2024-04-18T10:06:05.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e3555640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fa5dc15ab20 con 0x7fa5dc0b3150 2024-04-18T10:06:05.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e12f1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5dc0b4aa0 0x7fa5dc0b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e3555640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fa5dc15ae20 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f0cc966c640 1 -- 172.21.15.62:0/2647919618 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cc4146eb0 con 0x7f0cc4146900 2024-04-18T10:06:05.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f0cc966c640 1 -- 172.21.15.62:0/2647919618 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cc4149700 con 0x7f0cc4146900 2024-04-18T10:06:05.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b7b59e640 1 -- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b74035560 msgr2=0x7f2b740bb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b7b59e640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b74035560 0x7f2b740bb340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b7b59e640 1 -- 172.21.15.62:0/291389020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b740c8600 msgr2=0x7f2b740c1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b7b59e640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b740c8600 0x7f2b740c1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b7b59e640 1 -- 172.21.15.62:0/291389020 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b740c0d20 con 0x7f2b740c4520 2024-04-18T10:06:05.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fd8467fc640 1 -- 172.21.15.62:0/4242478063 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0ae050 con 0x7fd848073aa0 2024-04-18T10:06:05.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b7b59e640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b740c4520 0x7f2b740c32a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00b3e0 tx=0x7f2b6c00b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5efa4640 1 -- start start 2024-04-18T10:06:05.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f3cca7fc640 1 -- 172.21.15.62:0/1283845604 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0015870 con 0x7f3cd4142150 2024-04-18T10:06:05.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f3cca7fc640 1 -- 172.21.15.62:0/1283845604 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd0015a10 con 0x7f3cd4142150 2024-04-18T10:06:05.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f3cca7fc640 1 -- 172.21.15.62:0/1283845604 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0015dc0 con 0x7f3cd4142150 2024-04-18T10:06:05.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f3cdaaed640 1 -- 172.21.15.62:0/1283845604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cd41419f0 con 0x7f3cd4142150 2024-04-18T10:06:05.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f3cdaaed640 1 -- 172.21.15.62:0/1283845604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cd4141710 con 0x7f3cd4142150 2024-04-18T10:06:05.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5efa4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6d5806daf0 0x7f6d58143f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b637fe640 1 -- 172.21.15.62:0/291389020 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c015020 con 0x7f2b740c4520 2024-04-18T10:06:05.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b637fe640 1 -- 172.21.15.62:0/291389020 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b6c011850 con 0x7f2b740c4520 2024-04-18T10:06:05.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b637fe640 1 -- 172.21.15.62:0/291389020 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c011c40 con 0x7f2b740c4520 2024-04-18T10:06:05.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5dfa2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6d5806daf0 0x7f6d58143f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fd84ffef640 1 -- 172.21.15.62:0/4242478063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd84813de10 con 0x7fd848073aa0 2024-04-18T10:06:05.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fd84ffef640 1 -- 172.21.15.62:0/4242478063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd84813ff60 con 0x7fd848073aa0 2024-04-18T10:06:05.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5dfa2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f6d5806daf0 0x7f6d58143f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53392/0 (socket says 172.21.15.62:53392) 2024-04-18T10:06:05.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5dfa2640 1 -- 172.21.15.62:0/4256996025 learned_addr learned my addr 172.21.15.62:0/4256996025 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fd8467fc640 1 -- 172.21.15.62:0/4242478063 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd83c0b8860 con 0x7fd848073aa0 2024-04-18T10:06:05.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fd8467fc640 1 -- 172.21.15.62:0/4242478063 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0b8c50 con 0x7fd848073aa0 2024-04-18T10:06:05.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f627ffff640 1 -- 172.21.15.62:0/822914824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62800b7e70 con 0x7f628013e1e0 2024-04-18T10:06:05.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f627ffff640 1 -- 172.21.15.62:0/822914824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62801424f0 con 0x7f628013e1e0 2024-04-18T10:06:05.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b81165640 1 -- 172.21.15.62:0/291389020 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b740c5af0 con 0x7f2b740c4520 2024-04-18T10:06:05.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b81165640 1 -- 172.21.15.62:0/291389020 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b740c24f0 con 0x7f2b740c4520 2024-04-18T10:06:05.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f38dffff640 1 -- 172.21.15.62:0/1180345932 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f38e80071f0 con 0x7f38f80b0c60 2024-04-18T10:06:05.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f38dffff640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f38e4068450 0x7f38e406a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f38dffff640 1 -- 172.21.15.62:0/1180345932 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f38e8085cc0 con 0x7f38f80b0c60 2024-04-18T10:06:05.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e12f1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5dc0b4aa0 0x7fa5dc0b0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:47954/0 (socket says 172.21.15.62:47954) 2024-04-18T10:06:05.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e12f1640 1 -- 172.21.15.62:0/3075873738 learned_addr learned my addr 172.21.15.62:0/3075873738 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e12f1640 1 -- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5dc0b3150 msgr2=0x7fa5dc073220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-18T10:06:05.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e12f1640 1 --2- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5dc0b3150 0x7fa5dc073220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e12f1640 1 -- 172.21.15.62:0/3075873738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5dc0b8dc0 msgr2=0x7fa5dc1eddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e12f1640 1 --2- 172.21.15.62:0/3075873738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5dc0b8dc0 0x7fa5dc1eddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e12f1640 1 -- 172.21.15.62:0/3075873738 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5dc0b3680 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5efa4640 1 --2- 172.21.15.62:0/4256996025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d580733d0 0x7f6d5814e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5efa4640 1 -- 172.21.15.62:0/4256996025 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d580b5ea0 con 0x7f6d58144460 2024-04-18T10:06:05.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5efa4640 1 -- 172.21.15.62:0/4256996025 --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f6d580b6110 con 0x7f6d5806daf0 2024-04-18T10:06:05.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5efa4640 1 -- 172.21.15.62:0/4256996025 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f6d5806ded0 con 0x7f6d580733d0 2024-04-18T10:06:05.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5dfa2640 1 -- 172.21.15.62:0/4256996025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d580733d0 msgr2=0x7f6d5814e7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-18T10:06:05.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5dfa2640 1 --2- 172.21.15.62:0/4256996025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d580733d0 0x7f6d5814e7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5dfa2640 1 -- 172.21.15.62:0/4256996025 >> v1:172.21.15.145:6789/0 conn(0x7f6d58144460 legacy=0x7f6d58072cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-18T10:06:05.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f6d5dfa2640 1 -- 172.21.15.62:0/4256996025 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d580af390 con 0x7f6d5806daf0 2024-04-18T10:06:05.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fd84ffef640 1 -- 172.21.15.62:0/4242478063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8280052a0 con 0x7fd848073aa0 2024-04-18T10:06:05.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f3cdaaed640 1 -- 172.21.15.62:0/1283845604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cbc0052a0 con 0x7f3cd4142150 2024-04-18T10:06:05.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7f2b81165640 1 -- 172.21.15.62:0/291389020 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b740053a0 con 0x7f2b740c4520 2024-04-18T10:06:05.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.314+0000 7fa5e12f1640 1 --2- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5dc0b4aa0 0x7fa5dc0b0de0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc0ae040 tx=0x7fa5cc019290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f38f67fc640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f38e4068450 0x7f38e406a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f0cc966c640 1 -- 172.21.15.62:0/2647919618 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cc404fe10 con 0x7f0cc4146900 2024-04-18T10:06:05.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f627ffff640 1 -- 172.21.15.62:0/822914824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f628004fe10 con 0x7f628013e1e0 2024-04-18T10:06:05.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7fd8467fc640 1 -- 172.21.15.62:0/4242478063 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd83c0b1070 con 0x7fd848073aa0 2024-04-18T10:06:05.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7fa5d1ffb640 1 -- 172.21.15.62:0/3075873738 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc0b0040 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7fa5e3555640 1 -- 172.21.15.62:0/3075873738 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5dc1f1790 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7fa5e3555640 1 -- 172.21.15.62:0/3075873738 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5dc14cf10 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7fd8467fc640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd8340683d0 0x7fd83406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f2b637fe640 1 -- 172.21.15.62:0/291389020 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2b6c018020 con 0x7f2b740c4520 2024-04-18T10:06:05.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f6267fff640 1 -- 172.21.15.62:0/822914824 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f627001a070 con 0x7f628013e1e0 2024-04-18T10:06:05.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f38f67fc640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f38e4068450 0x7f38e406a910 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f38f813b0d0 tx=0x7f38ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7fd84d58a640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd8340683d0 0x7fd83406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f6d5dfa2640 1 --2- 172.21.15.62:0/4256996025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5806daf0 0x7f6d58143f20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6d54009ed0 tx=0x7f6d540251e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9b7f11f6fcb60a server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f6267fff640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f626c0683d0 0x7f626c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f2b637fe640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2b68068450 0x7f2b6806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f2b637fe640 1 -- 172.21.15.62:0/291389020 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2b6c086c10 con 0x7f2b740c4520 2024-04-18T10:06:05.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f0cc0ff9640 1 -- 172.21.15.62:0/2647919618 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0cbc01b020 con 0x7f0cc4146900 2024-04-18T10:06:05.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7fa5e3555640 1 -- 172.21.15.62:0/3075873738 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5dc1f2250 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f2b7ad9d640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2b68068450 0x7f2b6806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f627e7fc640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f626c0683d0 0x7f626c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7fd84d58a640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd8340683d0 0x7fd83406a890 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd8481dd700 tx=0x7fd840008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f0cc0ff9640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0cb0068380 0x7f0cb006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f0cc0ff9640 1 -- 172.21.15.62:0/2647919618 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0cbc0870f0 con 0x7f0cc4146900 2024-04-18T10:06:05.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f0cc3fff640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0cb0068380 0x7f0cb006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f2b7ad9d640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2b68068450 0x7f2b6806a910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0510b0 tx=0x7f2b7c06e040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.318+0000 7f627e7fc640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f626c0683d0 0x7f626c06a890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6280138360 tx=0x7f6278005d40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.322+0000 7f0cc3fff640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0cb0068380 0x7f0cb006a840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0cc414b5d0 tx=0x7f0cb4003700 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.322+0000 7fd8467fc640 1 -- 172.21.15.62:0/4242478063 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd83c0f3440 con 0x7fd848073aa0 2024-04-18T10:06:05.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.322+0000 7f6267fff640 1 -- 172.21.15.62:0/822914824 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6270054b10 con 0x7f628013e1e0 2024-04-18T10:06:05.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.322+0000 7f38dffff640 1 -- 172.21.15.62:0/1180345932 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f38e8055c20 con 0x7f38f80b0c60 2024-04-18T10:06:05.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.322+0000 7f6d5cfa0640 1 -- 172.21.15.62:0/4256996025 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d540b9070 con 0x7f6d5806daf0 2024-04-18T10:06:05.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.322+0000 7f3cca7fc640 1 -- 172.21.15.62:0/1283845604 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3cd000f480 con 0x7f3cd4142150 2024-04-18T10:06:05.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.326+0000 7f3cca7fc640 1 --2- 172.21.15.62:0/1283845604 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3cb80683d0 0x7f3cb806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.326+0000 7f3cd92ea640 1 --2- 172.21.15.62:0/1283845604 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3cb80683d0 0x7f3cb806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.326+0000 7f3cd92ea640 1 --2- 172.21.15.62:0/1283845604 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3cb80683d0 0x7f3cb806a890 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0ae000 tx=0x7f3ccc01b570 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.326+0000 7fd8467fc640 1 -- 172.21.15.62:0/4242478063 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd83c0cf080 con 0x7fd848073aa0 2024-04-18T10:06:05.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.330+0000 7fa5d1ffb640 1 -- 172.21.15.62:0/3075873738 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5cc0b9070 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.330+0000 7fa5d1ffb640 1 -- 172.21.15.62:0/3075873738 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc0b4860 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.330+0000 7f2b637fe640 1 -- 172.21.15.62:0/291389020 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2b6c01f020 con 0x7f2b740c4520 2024-04-18T10:06:05.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.338+0000 7f6267fff640 1 -- 172.21.15.62:0/822914824 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f627000ae20 con 0x7f628013e1e0 2024-04-18T10:06:05.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.338+0000 7f3cca7fc640 1 -- 172.21.15.62:0/1283845604 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3cd0055350 con 0x7f3cd4142150 2024-04-18T10:06:05.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5cfa0640 1 -- 172.21.15.62:0/4256996025 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d540b4400 con 0x7f6d5806daf0 2024-04-18T10:06:05.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5cfa0640 1 -- 172.21.15.62:0/4256996025 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d540b47d0 con 0x7f6d5806daf0 2024-04-18T10:06:05.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5efa4640 1 -- 172.21.15.62:0/4256996025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5806daf0 msgr2=0x7f6d58143f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5efa4640 1 --2- 172.21.15.62:0/4256996025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5806daf0 0x7f6d58143f20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6d54009ed0 tx=0x7f6d540251e0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5efa4640 1 -- 172.21.15.62:0/4256996025 shutdown_connections 2024-04-18T10:06:05.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5efa4640 1 --2- 172.21.15.62:0/4256996025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d580733d0 0x7f6d5814e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5efa4640 1 --2- 172.21.15.62:0/4256996025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5806daf0 0x7f6d58143f20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5efa4640 1 -- 172.21.15.62:0/4256996025 >> 172.21.15.62:0/4256996025 conn(0x7f6d580b9ae0 msgr2=0x7f6d580b9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5efa4640 1 -- 172.21.15.62:0/4256996025 shutdown_connections 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5efa4640 1 -- 172.21.15.62:0/4256996025 wait complete. 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5efa4640 1 Processor -- start 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5efa4640 1 -- start start 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5efa4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5806daf0 0x7f6d58147850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.342+0000 7f6d5efa4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d580733d0 0x7f6d5813e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5efa4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d58144460 0x7f6d5813e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5efa4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6d580aebc0 con 0x7f6d5806daf0 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5efa4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f6d580aea40 con 0x7f6d580733d0 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5efa4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f6d58150d10 con 0x7f6d58144460 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5dfa2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5806daf0 0x7f6d58147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5dfa2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5806daf0 0x7f6d58147850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53400/0 (socket says 172.21.15.62:53400) 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5dfa2640 1 -- 172.21.15.62:0/2213367236 learned_addr learned my addr 172.21.15.62:0/2213367236 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5d7a1640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d580733d0 0x7f6d5813e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5dfa2640 1 -- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d58144460 msgr2=0x7f6d5813e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5dfa2640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d58144460 0x7f6d5813e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5dfa2640 1 -- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d580733d0 msgr2=0x7f6d5813e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5dfa2640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d580733d0 0x7f6d5813e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5dfa2640 1 -- 172.21.15.62:0/2213367236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d581470d0 con 0x7f6d5806daf0 2024-04-18T10:06:05.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f0cc0ff9640 1 -- 172.21.15.62:0/2647919618 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0cbc057720 con 0x7f0cc4146900 2024-04-18T10:06:05.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7f6d5dfa2640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5806daf0 0x7f6d58147850 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d540b7040 tx=0x7f6d54021f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7fa5d1ffb640 1 -- 172.21.15.62:0/3075873738 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa5cc0b4a00 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.346+0000 7fa5d1ffb640 1 --2- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa5c00683b0 0x7fa5c006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.350+0000 7fa5d1ffb640 1 -- 172.21.15.62:0/3075873738 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa5cc121570 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.350+0000 7fa5e0af0640 1 --2- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa5c00683b0 0x7fa5c006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.350+0000 7fa5e0af0640 1 --2- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa5c00683b0 0x7fa5c006a870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8007fd0 tx=0x7fa5d8007940 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.350+0000 7f3cca7fc640 1 -- 172.21.15.62:0/1283845604 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3cd0019030 con 0x7f3cd4142150 2024-04-18T10:06:05.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.350+0000 7f6d4e7fc640 1 -- 172.21.15.62:0/2213367236 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d540c3070 con 0x7f6d5806daf0 2024-04-18T10:06:05.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.350+0000 7f6d4e7fc640 1 -- 172.21.15.62:0/2213367236 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d540b4e30 con 0x7f6d5806daf0 2024-04-18T10:06:05.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.350+0000 7f6d5efa4640 1 -- 172.21.15.62:0/2213367236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d58145a50 con 0x7f6d5806daf0 2024-04-18T10:06:05.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.350+0000 7f6d4e7fc640 1 -- 172.21.15.62:0/2213367236 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d540b9040 con 0x7f6d5806daf0 2024-04-18T10:06:05.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.350+0000 7f6d5efa4640 1 -- 172.21.15.62:0/2213367236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d58146df0 con 0x7f6d5806daf0 2024-04-18T10:06:05.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.354+0000 7f6d5efa4640 1 -- 172.21.15.62:0/2213367236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d5804fe10 con 0x7f6d5806daf0 2024-04-18T10:06:05.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.354+0000 7f6d4e7fc640 1 -- 172.21.15.62:0/2213367236 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6d540c6070 con 0x7f6d5806daf0 2024-04-18T10:06:05.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.354+0000 7f6d4e7fc640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6d3c068400 0x7f6d3c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.358+0000 7f6d5d7a1640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6d3c068400 0x7f6d3c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.358+0000 7f6d4e7fc640 1 -- 172.21.15.62:0/2213367236 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6d540f83e0 con 0x7f6d5806daf0 2024-04-18T10:06:05.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.358+0000 7f6d5d7a1640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6d3c068400 0x7f6d3c06a8c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6d58143950 tx=0x7f6d48008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.358+0000 7fa5d1ffb640 1 -- 172.21.15.62:0/3075873738 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa5dc1f2250 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.366+0000 7f6d4e7fc640 1 -- 172.21.15.62:0/2213367236 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6d5401ee00 con 0x7f6d5806daf0 2024-04-18T10:06:05.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.630+0000 7f38fd86d640 1 -- 172.21.15.62:0/1180345932 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f38d8005590 con 0x7f38f80b0c60 2024-04-18T10:06:05.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.630+0000 7f38dffff640 1 -- 172.21.15.62:0/1180345932 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f38e8059f20 con 0x7f38f80b0c60 2024-04-18T10:06:05.630 INFO:teuthology.orchestra.run.smithi062.stdout:55834574855 2024-04-18T10:06:05.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.634+0000 7f38ddffb640 1 -- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f38e4068450 msgr2=0x7f38e406a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.634+0000 7f38ddffb640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f38e4068450 0x7f38e406a910 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f38f813b0d0 tx=0x7f38ec008040 comp rx=0 tx=0).stop 2024-04-18T10:06:05.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.634+0000 7f38ddffb640 1 -- 172.21.15.62:0/1180345932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38f80b0c60 msgr2=0x7f38f81e4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.634+0000 7f38ddffb640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38f80b0c60 0x7f38f81e4230 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f38e800c500 tx=0x7f38e800c9c0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.634+0000 7f38ddffb640 1 -- 172.21.15.62:0/1180345932 shutdown_connections 2024-04-18T10:06:05.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.634+0000 7f38ddffb640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f38e4068450 0x7f38e406a910 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.634+0000 7f38ddffb640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38f814c520 0x7f38f81e4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.634+0000 7f38ddffb640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38f81458a0 0x7f38f813da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.634+0000 7f38ddffb640 1 --2- 172.21.15.62:0/1180345932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38f80b0c60 0x7f38f81e4230 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.634+0000 7f38ddffb640 1 -- 172.21.15.62:0/1180345932 >> 172.21.15.62:0/1180345932 conn(0x7f38f80b92f0 msgr2=0x7f38f8141830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.634+0000 7f38ddffb640 1 -- 172.21.15.62:0/1180345932 shutdown_connections 2024-04-18T10:06:05.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.638+0000 7f38ddffb640 1 -- 172.21.15.62:0/1180345932 wait complete. 2024-04-18T10:06:05.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.646+0000 7f2b81165640 1 -- 172.21.15.62:0/291389020 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f2b7403d190 con 0x7f2b740c4520 2024-04-18T10:06:05.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.646+0000 7f2b637fe640 1 -- 172.21.15.62:0/291389020 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2b6c056bf0 con 0x7f2b740c4520 2024-04-18T10:06:05.644 INFO:teuthology.orchestra.run.smithi062.stdout:55834574854 2024-04-18T10:06:05.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.650+0000 7f2b617fa640 1 -- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2b68068450 msgr2=0x7f2b6806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.650+0000 7f2b617fa640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2b68068450 0x7f2b6806a910 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c0510b0 tx=0x7f2b7c06e040 comp rx=0 tx=0).stop 2024-04-18T10:06:05.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.650+0000 7f2b617fa640 1 -- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b740c4520 msgr2=0x7f2b740c32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.650+0000 7f2b617fa640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b740c4520 0x7f2b740c32a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00b3e0 tx=0x7f2b6c00b8a0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.650+0000 7f2b617fa640 1 -- 172.21.15.62:0/291389020 shutdown_connections 2024-04-18T10:06:05.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.650+0000 7f2b617fa640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2b68068450 0x7f2b6806a910 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.650+0000 7f2b617fa640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2b740c4520 0x7f2b740c32a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.650+0000 7f2b617fa640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2b740c8600 0x7f2b740c1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.650+0000 7f2b617fa640 1 --2- 172.21.15.62:0/291389020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2b74035560 0x7f2b740bb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.650+0000 7f2b617fa640 1 -- 172.21.15.62:0/291389020 >> 172.21.15.62:0/291389020 conn(0x7f2b7403abd0 msgr2=0x7f2b740055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.650+0000 7f2b617fa640 1 -- 172.21.15.62:0/291389020 shutdown_connections 2024-04-18T10:06:05.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.650+0000 7f2b617fa640 1 -- 172.21.15.62:0/291389020 wait complete. 2024-04-18T10:06:05.658 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2024-04-18T10:06:05.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.658+0000 7fd84ffef640 1 -- 172.21.15.62:0/4242478063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd828005860 con 0x7fd848073aa0 2024-04-18T10:06:05.659 DEBUG:teuthology.parallel:result is None 2024-04-18T10:06:05.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.662+0000 7fd8467fc640 1 -- 172.21.15.62:0/4242478063 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd83c0bc030 con 0x7fd848073aa0 2024-04-18T10:06:05.659 INFO:teuthology.orchestra.run.smithi062.stdout:55834574855 2024-04-18T10:06:05.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.662+0000 7fd84ffef640 1 -- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd8340683d0 msgr2=0x7fd83406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.662+0000 7fd84ffef640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd8340683d0 0x7fd83406a890 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd8481dd700 tx=0x7fd840008040 comp rx=0 tx=0).stop 2024-04-18T10:06:05.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.662+0000 7fd84ffef640 1 -- 172.21.15.62:0/4242478063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd848073aa0 msgr2=0x7fd848141bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.662+0000 7fd84ffef640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd848073aa0 0x7fd848141bd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd83c01ccf0 tx=0x7fd83c01b3e0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.662+0000 7fd84ffef640 1 -- 172.21.15.62:0/4242478063 shutdown_connections 2024-04-18T10:06:05.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.666+0000 7fd84ffef640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd8340683d0 0x7fd83406a890 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.666+0000 7fd84ffef640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8481e0080 0x7fd8481d5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.666+0000 7fd84ffef640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8480b1880 0x7fd8481dfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.666+0000 7fd84ffef640 1 --2- 172.21.15.62:0/4242478063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd848073aa0 0x7fd848141bd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.666+0000 7fd84ffef640 1 -- 172.21.15.62:0/4242478063 >> 172.21.15.62:0/4242478063 conn(0x7fd8480b92f0 msgr2=0x7fd8480b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.666+0000 7fd84ffef640 1 -- 172.21.15.62:0/4242478063 shutdown_connections 2024-04-18T10:06:05.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.666+0000 7fd84ffef640 1 -- 172.21.15.62:0/4242478063 wait complete. 2024-04-18T10:06:05.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.670+0000 7f627ffff640 1 -- 172.21.15.62:0/822914824 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f62800b52e0 con 0x7f628013e1e0 2024-04-18T10:06:05.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.674+0000 7f6267fff640 1 -- 172.21.15.62:0/822914824 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6270056a70 con 0x7f628013e1e0 2024-04-18T10:06:05.671 INFO:teuthology.orchestra.run.smithi062.stdout:55834574856 2024-04-18T10:06:05.673 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2024-04-18T10:06:05.673 DEBUG:teuthology.parallel:result is None 2024-04-18T10:06:05.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.674+0000 7fa5e3555640 1 -- 172.21.15.62:0/3075873738 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fa5dc04fe10 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.674+0000 7f627ffff640 1 -- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f626c0683d0 msgr2=0x7f626c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.674+0000 7f627ffff640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f626c0683d0 0x7f626c06a890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6280138360 tx=0x7f6278005d40 comp rx=0 tx=0).stop 2024-04-18T10:06:05.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.674+0000 7f627ffff640 1 -- 172.21.15.62:0/822914824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f628013e1e0 msgr2=0x7f62801df830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.674+0000 7f627ffff640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f628013e1e0 0x7f62801df830 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f627000cb10 tx=0x7f627000a060 comp rx=0 tx=0).stop 2024-04-18T10:06:05.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.674+0000 7f627ffff640 1 -- 172.21.15.62:0/822914824 shutdown_connections 2024-04-18T10:06:05.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.674+0000 7f627ffff640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f626c0683d0 0x7f626c06a890 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.674+0000 7f627ffff640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62801e02b0 0x7f628012c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.674+0000 7f627ffff640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62801443f0 0x7f62801dfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.674+0000 7f627ffff640 1 --2- 172.21.15.62:0/822914824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f628013e1e0 0x7f62801df830 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.674+0000 7f627ffff640 1 -- 172.21.15.62:0/822914824 >> 172.21.15.62:0/822914824 conn(0x7f62800b91f0 msgr2=0x7f628013c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7f627ffff640 1 -- 172.21.15.62:0/822914824 shutdown_connections 2024-04-18T10:06:05.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7f627ffff640 1 -- 172.21.15.62:0/822914824 wait complete. 2024-04-18T10:06:05.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5d1ffb640 1 -- 172.21.15.62:0/3075873738 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa5cc11fe20 con 0x7fa5dc0b4aa0 2024-04-18T10:06:05.677 INFO:teuthology.orchestra.run.smithi062.stdout:55834574856 2024-04-18T10:06:05.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5e3555640 1 -- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa5c00683b0 msgr2=0x7fa5c006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5e3555640 1 --2- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa5c00683b0 0x7fa5c006a870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8007fd0 tx=0x7fa5d8007940 comp rx=0 tx=0).stop 2024-04-18T10:06:05.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5e3555640 1 -- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5dc0b4aa0 msgr2=0x7fa5dc0b0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5e3555640 1 --2- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5dc0b4aa0 0x7fa5dc0b0de0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc0ae040 tx=0x7fa5cc019290 comp rx=0 tx=0).stop 2024-04-18T10:06:05.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5e3555640 1 -- 172.21.15.62:0/3075873738 shutdown_connections 2024-04-18T10:06:05.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5e3555640 1 --2- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa5c00683b0 0x7fa5c006a870 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5e3555640 1 --2- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5dc0b3150 0x7fa5dc073220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5e3555640 1 --2- 172.21.15.62:0/3075873738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5dc0b8dc0 0x7fa5dc1eddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5e3555640 1 --2- 172.21.15.62:0/3075873738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5dc0b4aa0 0x7fa5dc0b0de0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5e3555640 1 -- 172.21.15.62:0/3075873738 >> 172.21.15.62:0/3075873738 conn(0x7fa5dc09b530 msgr2=0x7fa5dc09a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5e3555640 1 -- 172.21.15.62:0/3075873738 shutdown_connections 2024-04-18T10:06:05.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7f0cc966c640 1 -- 172.21.15.62:0/2647919618 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0cc4147c10 con 0x7f0cc4146900 2024-04-18T10:06:05.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7fa5e3555640 1 -- 172.21.15.62:0/3075873738 wait complete. 2024-04-18T10:06:05.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.678+0000 7f0cc0ff9640 1 -- 172.21.15.62:0/2647919618 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0cbc057020 con 0x7f0cc4146900 2024-04-18T10:06:05.680 INFO:teuthology.orchestra.run.smithi062.stdout:55834574854 2024-04-18T10:06:05.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 -- 172.21.15.62:0/1283845604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f3cbc005860 con 0x7f3cd4142150 2024-04-18T10:06:05.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cca7fc640 1 -- 172.21.15.62:0/1283845604 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3cd00579b0 con 0x7f3cd4142150 2024-04-18T10:06:05.681 INFO:teuthology.orchestra.run.smithi062.stdout:55834574856 2024-04-18T10:06:05.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f0cc966c640 1 -- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0cb0068380 msgr2=0x7f0cb006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f0cc966c640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0cb0068380 0x7f0cb006a840 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0cc414b5d0 tx=0x7f0cb4003700 comp rx=0 tx=0).stop 2024-04-18T10:06:05.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f0cc966c640 1 -- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146900 msgr2=0x7f0cc413c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f0cc966c640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146900 0x7f0cc413c340 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc007d20 tx=0x7f0cbc00e4b0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 -- 172.21.15.62:0/1283845604 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3cb80683d0 msgr2=0x7f3cb806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 --2- 172.21.15.62:0/1283845604 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3cb80683d0 0x7f3cb806a890 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0ae000 tx=0x7f3ccc01b570 comp rx=0 tx=0).stop 2024-04-18T10:06:05.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 -- 172.21.15.62:0/1283845604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4142150 msgr2=0x7f3cd4142530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 --2- 172.21.15.62:0/1283845604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4142150 0x7f3cd4142530 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3cd0011610 tx=0x7f3cd0011ad0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f0cc966c640 1 -- 172.21.15.62:0/2647919618 shutdown_connections 2024-04-18T10:06:05.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f0cc966c640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0cb0068380 0x7f0cb006a840 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f0cc966c640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4146900 0x7f0cc413c340 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f0cc966c640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cc40b22a0 0x7f0cc4147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f0cc966c640 1 --2- 172.21.15.62:0/2647919618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cc40b1ec0 0x7f0cc413fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f0cc966c640 1 -- 172.21.15.62:0/2647919618 >> 172.21.15.62:0/2647919618 conn(0x7f0cc40b91f0 msgr2=0x7f0cc40b7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 -- 172.21.15.62:0/1283845604 shutdown_connections 2024-04-18T10:06:05.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 --2- 172.21.15.62:0/1283845604 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3cb80683d0 0x7f3cb806a890 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 --2- 172.21.15.62:0/1283845604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cd4148e90 0x7f3cd41e84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 --2- 172.21.15.62:0/1283845604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cd4148a30 0x7f3cd41e7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 --2- 172.21.15.62:0/1283845604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cd4142150 0x7f3cd4142530 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 -- 172.21.15.62:0/1283845604 >> 172.21.15.62:0/1283845604 conn(0x7f3cd40ba370 msgr2=0x7f3cd40b4890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f0cc966c640 1 -- 172.21.15.62:0/2647919618 shutdown_connections 2024-04-18T10:06:05.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 -- 172.21.15.62:0/1283845604 shutdown_connections 2024-04-18T10:06:05.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f3cdaaed640 1 -- 172.21.15.62:0/1283845604 wait complete. 2024-04-18T10:06:05.686 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.682+0000 7f0cc966c640 1 -- 172.21.15.62:0/2647919618 wait complete. 2024-04-18T10:06:05.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.694+0000 7f6d5efa4640 1 -- 172.21.15.62:0/2213367236 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f6d580afc80 con 0x7f6d5806daf0 2024-04-18T10:06:05.695 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2024-04-18T10:06:05.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d4e7fc640 1 -- 172.21.15.62:0/2213367236 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6d540faa40 con 0x7f6d5806daf0 2024-04-18T10:06:05.696 INFO:teuthology.orchestra.run.smithi062.stdout:55834574856 2024-04-18T10:06:05.696 DEBUG:teuthology.parallel:result is None 2024-04-18T10:06:05.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d5efa4640 1 -- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6d3c068400 msgr2=0x7f6d3c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d5efa4640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6d3c068400 0x7f6d3c06a8c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6d58143950 tx=0x7f6d48008040 comp rx=0 tx=0).stop 2024-04-18T10:06:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d5efa4640 1 -- 172.21.15.62:0/2213367236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5806daf0 msgr2=0x7f6d58147850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d5efa4640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5806daf0 0x7f6d58147850 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6d540b7040 tx=0x7f6d54021f80 comp rx=0 tx=0).stop 2024-04-18T10:06:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d5efa4640 1 -- 172.21.15.62:0/2213367236 shutdown_connections 2024-04-18T10:06:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d5efa4640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6d3c068400 0x7f6d3c06a8c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d5efa4640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d58144460 0x7f6d5813e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d5efa4640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d580733d0 0x7f6d5813e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d5efa4640 1 --2- 172.21.15.62:0/2213367236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d5806daf0 0x7f6d58147850 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d5efa4640 1 -- 172.21.15.62:0/2213367236 >> 172.21.15.62:0/2213367236 conn(0x7f6d580b9ae0 msgr2=0x7f6d580b6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d5efa4640 1 -- 172.21.15.62:0/2213367236 shutdown_connections 2024-04-18T10:06:05.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.698+0000 7f6d5efa4640 1 -- 172.21.15.62:0/2213367236 wait complete. 2024-04-18T10:06:05.699 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2024-04-18T10:06:05.699 DEBUG:teuthology.parallel:result is None 2024-04-18T10:06:05.704 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-04-18T10:06:05.705 DEBUG:teuthology.parallel:result is None 2024-04-18T10:06:05.714 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2024-04-18T10:06:05.714 DEBUG:teuthology.parallel:result is None 2024-04-18T10:06:05.715 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-04-18T10:06:05.715 DEBUG:teuthology.parallel:result is None 2024-04-18T10:06:05.717 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2024-04-18T10:06:05.717 DEBUG:teuthology.parallel:result is None 2024-04-18T10:06:05.717 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-18T10:06:05.717 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T10:06:05.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161c3f8640 1 Processor -- start 2024-04-18T10:06:05.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161c3f8640 1 -- start start 2024-04-18T10:06:05.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161c3f8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f16140b6260 0x7f1614148260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161c3f8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16140b7850 con 0x7f16140b6640 2024-04-18T10:06:05.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161c3f8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16140b4c20 con 0x7f16141487a0 2024-04-18T10:06:05.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161c3f8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f1614148cb0 con 0x7f16140b6260 2024-04-18T10:06:05.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161a995640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f16140b6640 0x7f16140b5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54974/0 (socket says 172.21.15.62:54974) 2024-04-18T10:06:05.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161a995640 1 -- 172.21.15.62:0/1630425892 learned_addr learned my addr 172.21.15.62:0/1630425892 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:05.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161a194640 1 --2- 172.21.15.62:0/1630425892 >> v2:172.21.15.62:3300/0 conn(0x7f16140b6260 0x7f1614148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f1619192640 1 -- 172.21.15.62:0/1630425892 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 561481580 0 0) 0x7f16140b7850 con 0x7f16140b6640 2024-04-18T10:06:05.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f1619192640 1 -- 172.21.15.62:0/1630425892 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1600003530 con 0x7f16140b6640 2024-04-18T10:06:05.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161a194640 1 -- 172.21.15.62:0/1630425892 >> v1:172.21.15.145:6789/0 conn(0x7f16141487a0 legacy=0x7f16141419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.865 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161a194640 1 -- 172.21.15.62:0/1630425892 >> v1:172.21.15.62:6789/0 conn(0x7f16140b6640 legacy=0x7f16140b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.866 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161a194640 1 -- 172.21.15.62:0/1630425892 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16141420b0 con 0x7f16140b6260 2024-04-18T10:06:05.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.866+0000 7f161a194640 1 --2- 172.21.15.62:0/1630425892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16140b6260 0x7f1614148260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f16100097c0 tx=0x7f161001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e63f0368cd10cbb server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f1619192640 1 -- 172.21.15.62:0/1630425892 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16100b0070 con 0x7f16140b6260 2024-04-18T10:06:05.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f1619192640 1 -- 172.21.15.62:0/1630425892 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16100ab600 con 0x7f16140b6260 2024-04-18T10:06:05.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f1619192640 1 -- 172.21.15.62:0/1630425892 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16100ab9d0 con 0x7f16140b6260 2024-04-18T10:06:05.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 -- 172.21.15.62:0/1630425892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16140b6260 msgr2=0x7f1614148260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 --2- 172.21.15.62:0/1630425892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16140b6260 0x7f1614148260 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f16100097c0 tx=0x7f161001c380 comp rx=0 tx=0).stop 2024-04-18T10:06:05.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 -- 172.21.15.62:0/1630425892 shutdown_connections 2024-04-18T10:06:05.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 --2- 172.21.15.62:0/1630425892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16140b6260 0x7f1614148260 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 -- 172.21.15.62:0/1630425892 >> 172.21.15.62:0/1630425892 conn(0x7f16140bd5c0 msgr2=0x7f16140bd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:05.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 -- 172.21.15.62:0/1630425892 shutdown_connections 2024-04-18T10:06:05.868 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 -- 172.21.15.62:0/1630425892 wait complete. 2024-04-18T10:06:05.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 Processor -- start 2024-04-18T10:06:05.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 -- start start 2024-04-18T10:06:05.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140b6260 0x7f16141e3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16140b6640 0x7f161413fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16141487a0 0x7f1614140150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1614156250 con 0x7f16141487a0 2024-04-18T10:06:05.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f16141560d0 con 0x7f16140b6640 2024-04-18T10:06:05.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f16141563d0 con 0x7f16140b6260 2024-04-18T10:06:05.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161a995640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16141487a0 0x7f1614140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161a995640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16141487a0 0x7f1614140150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53416/0 (socket says 172.21.15.62:53416) 2024-04-18T10:06:05.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161a995640 1 -- 172.21.15.62:0/1372964784 learned_addr learned my addr 172.21.15.62:0/1372964784 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:05.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161a995640 1 -- 172.21.15.62:0/1372964784 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140b6260 msgr2=0x7f16141e3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161a995640 1 --2- 172.21.15.62:0/1372964784 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140b6260 0x7f16141e3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161a995640 1 -- 172.21.15.62:0/1372964784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16140b6640 msgr2=0x7f161413fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:05.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161a995640 1 --2- 172.21.15.62:0/1372964784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16140b6640 0x7f161413fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:05.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161a995640 1 -- 172.21.15.62:0/1372964784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1614139f90 con 0x7f16141487a0 2024-04-18T10:06:05.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161a995640 1 --2- 172.21.15.62:0/1372964784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16141487a0 0x7f1614140150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f160c00ca70 tx=0x7f160c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f160affd640 1 -- 172.21.15.62:0/1372964784 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160c017070 con 0x7f16141487a0 2024-04-18T10:06:05.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f160affd640 1 -- 172.21.15.62:0/1372964784 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f160c00abd0 con 0x7f16141487a0 2024-04-18T10:06:05.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f160affd640 1 -- 172.21.15.62:0/1372964784 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f160c012400 con 0x7f16141487a0 2024-04-18T10:06:05.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.870+0000 7f161c3f8640 1 -- 172.21.15.62:0/1372964784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f161413a4e0 con 0x7f16141487a0 2024-04-18T10:06:05.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.874+0000 7f161c3f8640 1 -- 172.21.15.62:0/1372964784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f161413a200 con 0x7f16141487a0 2024-04-18T10:06:05.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.874+0000 7f161c3f8640 1 -- 172.21.15.62:0/1372964784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1614073bf0 con 0x7f16141487a0 2024-04-18T10:06:05.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.874+0000 7f160affd640 1 -- 172.21.15.62:0/1372964784 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f160c01a070 con 0x7f16141487a0 2024-04-18T10:06:05.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.874+0000 7f160affd640 1 --2- 172.21.15.62:0/1372964784 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f15f8068380 0x7f15f806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:05.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.874+0000 7f160affd640 1 -- 172.21.15.62:0/1372964784 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f160c0865b0 con 0x7f16141487a0 2024-04-18T10:06:05.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.874+0000 7f161a194640 1 --2- 172.21.15.62:0/1372964784 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f15f8068380 0x7f15f806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:05.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.874+0000 7f161a194640 1 --2- 172.21.15.62:0/1372964784 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f15f8068380 0x7f15f806a840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f16100ad000 tx=0x7f1610008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:05.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:05.878+0000 7f160affd640 1 -- 172.21.15.62:0/1372964784 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f160c056510 con 0x7f16141487a0 2024-04-18T10:06:06.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.030+0000 7f161c3f8640 1 -- 172.21.15.62:0/1372964784 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1614068b90 con 0x7f15f8068380 2024-04-18T10:06:06.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.046+0000 7f160affd640 1 -- 172.21.15.62:0/1372964784 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f1614068b90 con 0x7f15f8068380 2024-04-18T10:06:06.049 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:06.050 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-18T10:06:06.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.050+0000 7f161c3f8640 1 -- 172.21.15.62:0/1372964784 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f15f8068380 msgr2=0x7f15f806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.050+0000 7f161c3f8640 1 --2- 172.21.15.62:0/1372964784 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f15f8068380 0x7f15f806a840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f16100ad000 tx=0x7f1610008040 comp rx=0 tx=0).stop 2024-04-18T10:06:06.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.050+0000 7f161c3f8640 1 -- 172.21.15.62:0/1372964784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16141487a0 msgr2=0x7f1614140150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.050+0000 7f161c3f8640 1 --2- 172.21.15.62:0/1372964784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16141487a0 0x7f1614140150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f160c00ca70 tx=0x7f160c00cf30 comp rx=0 tx=0).stop 2024-04-18T10:06:06.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.050+0000 7f161c3f8640 1 -- 172.21.15.62:0/1372964784 shutdown_connections 2024-04-18T10:06:06.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.050+0000 7f161c3f8640 1 --2- 172.21.15.62:0/1372964784 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f15f8068380 0x7f15f806a840 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.050+0000 7f161c3f8640 1 --2- 172.21.15.62:0/1372964784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16141487a0 0x7f1614140150 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.050+0000 7f161c3f8640 1 --2- 172.21.15.62:0/1372964784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f16140b6640 0x7f161413fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.050+0000 7f161c3f8640 1 --2- 172.21.15.62:0/1372964784 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f16140b6260 0x7f16141e3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.050+0000 7f161c3f8640 1 -- 172.21.15.62:0/1372964784 >> 172.21.15.62:0/1372964784 conn(0x7f16140bd5c0 msgr2=0x7f1614145f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:06.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.050+0000 7f161c3f8640 1 -- 172.21.15.62:0/1372964784 shutdown_connections 2024-04-18T10:06:06.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.050+0000 7f161c3f8640 1 -- 172.21.15.62:0/1372964784 wait complete. 2024-04-18T10:06:06.067 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-18T10:06:05.126999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219116,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755604,"statfs":{"total":773094113280,"available":772869738496,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.923340"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242380+0000","last_change":"2024-04-18T10:05:59.270038+0000","last_active":"2024-04-18T10:06:00.242380+0000","last_peered":"2024-04-18T10:06:00.242380+0000","last_clean":"2024-04-18T10:06:00.242380+0000","last_became_active":"2024-04-18T10:05:59.269226+0000","last_became_peered":"2024-04-18T10:05:59.269226+0000","last_unstale":"2024-04-18T10:06:00.242380+0000","last_undegraded":"2024-04-18T10:06:00.242380+0000","last_fullsized":"2024-04-18T10:06:00.242380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:45:43.670227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232794+0000","last_change":"2024-04-18T10:05:59.230274+0000","last_active":"2024-04-18T10:06:00.232794+0000","last_peered":"2024-04-18T10:06:00.232794+0000","last_clean":"2024-04-18T10:06:00.232794+0000","last_became_active":"2024-04-18T10:05:59.229843+0000","last_became_peered":"2024-04-18T10:05:59.229843+0000","last_unstale":"2024-04-18T10:06:00.232794+0000","last_undegraded":"2024-04-18T10:06:00.232794+0000","last_fullsized":"2024-04-18T10:06:00.232794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384628+0000","last_change":"2024-04-18T10:05:59.228858+0000","last_active":"2024-04-18T10:06:00.384628+0000","last_peered":"2024-04-18T10:06:00.384628+0000","last_clean":"2024-04-18T10:06:00.384628+0000","last_became_active":"2024-04-18T10:05:59.228382+0000","last_became_peered":"2024-04-18T10:05:59.228382+0000","last_unstale":"2024-04-18T10:06:00.384628+0000","last_undegraded":"2024-04-18T10:06:00.384628+0000","last_fullsized":"2024-04-18T10:06:00.384628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240458+0000","last_change":"2024-04-18T10:05:59.240142+0000","last_active":"2024-04-18T10:06:00.240458+0000","last_peered":"2024-04-18T10:06:00.240458+0000","last_clean":"2024-04-18T10:06:00.240458+0000","last_became_active":"2024-04-18T10:05:59.239298+0000","last_became_peered":"2024-04-18T10:05:59.239298+0000","last_unstale":"2024-04-18T10:06:00.240458+0000","last_undegraded":"2024-04-18T10:06:00.240458+0000","last_fullsized":"2024-04-18T10:06:00.240458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:49.354229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239594+0000","last_change":"2024-04-18T10:05:59.235434+0000","last_active":"2024-04-18T10:06:00.239594+0000","last_peered":"2024-04-18T10:06:00.239594+0000","last_clean":"2024-04-18T10:06:00.239594+0000","last_became_active":"2024-04-18T10:05:59.234575+0000","last_became_peered":"2024-04-18T10:05:59.234575+0000","last_unstale":"2024-04-18T10:06:00.239594+0000","last_undegraded":"2024-04-18T10:06:00.239594+0000","last_fullsized":"2024-04-18T10:06:00.239594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:34:28.871228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238452+0000","last_change":"2024-04-18T10:05:59.232083+0000","last_active":"2024-04-18T10:06:00.238452+0000","last_peered":"2024-04-18T10:06:00.238452+0000","last_clean":"2024-04-18T10:06:00.238452+0000","last_became_active":"2024-04-18T10:05:59.231313+0000","last_became_peered":"2024-04-18T10:05:59.231313+0000","last_unstale":"2024-04-18T10:06:00.238452+0000","last_undegraded":"2024-04-18T10:06:00.238452+0000","last_fullsized":"2024-04-18T10:06:00.238452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238876+0000","last_change":"2024-04-18T10:05:59.250962+0000","last_active":"2024-04-18T10:06:00.238876+0000","last_peered":"2024-04-18T10:06:00.238876+0000","last_clean":"2024-04-18T10:06:00.238876+0000","last_became_active":"2024-04-18T10:05:59.250059+0000","last_became_peered":"2024-04-18T10:05:59.250059+0000","last_unstale":"2024-04-18T10:06:00.238876+0000","last_undegraded":"2024-04-18T10:06:00.238876+0000","last_fullsized":"2024-04-18T10:06:00.238876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:12:17.167339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383437+0000","last_change":"2024-04-18T10:05:59.256210+0000","last_active":"2024-04-18T10:06:00.383437+0000","last_peered":"2024-04-18T10:06:00.383437+0000","last_clean":"2024-04-18T10:06:00.383437+0000","last_became_active":"2024-04-18T10:05:59.255975+0000","last_became_peered":"2024-04-18T10:05:59.255975+0000","last_unstale":"2024-04-18T10:06:00.383437+0000","last_undegraded":"2024-04-18T10:06:00.383437+0000","last_fullsized":"2024-04-18T10:06:00.383437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:49:34.764977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242471+0000","last_change":"2024-04-18T10:05:59.272589+0000","last_active":"2024-04-18T10:06:00.242471+0000","last_peered":"2024-04-18T10:06:00.242471+0000","last_clean":"2024-04-18T10:06:00.242471+0000","last_became_active":"2024-04-18T10:05:59.271883+0000","last_became_peered":"2024-04-18T10:05:59.271883+0000","last_unstale":"2024-04-18T10:06:00.242471+0000","last_undegraded":"2024-04-18T10:06:00.242471+0000","last_fullsized":"2024-04-18T10:06:00.242471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:24:15.929261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243230+0000","last_change":"2024-04-18T10:05:59.261146+0000","last_active":"2024-04-18T10:06:00.243230+0000","last_peered":"2024-04-18T10:06:00.243230+0000","last_clean":"2024-04-18T10:06:00.243230+0000","last_became_active":"2024-04-18T10:05:59.260455+0000","last_became_peered":"2024-04-18T10:05:59.260455+0000","last_unstale":"2024-04-18T10:06:00.243230+0000","last_undegraded":"2024-04-18T10:06:00.243230+0000","last_fullsized":"2024-04-18T10:06:00.243230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.244095+0000","last_change":"2024-04-18T10:05:59.260958+0000","last_active":"2024-04-18T10:06:00.244095+0000","last_peered":"2024-04-18T10:06:00.244095+0000","last_clean":"2024-04-18T10:06:00.244095+0000","last_became_active":"2024-04-18T10:05:59.260251+0000","last_became_peered":"2024-04-18T10:05:59.260251+0000","last_unstale":"2024-04-18T10:06:00.244095+0000","last_undegraded":"2024-04-18T10:06:00.244095+0000","last_fullsized":"2024-04-18T10:06:00.244095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384238+0000","last_change":"2024-04-18T10:05:59.238407+0000","last_active":"2024-04-18T10:06:00.384238+0000","last_peered":"2024-04-18T10:06:00.384238+0000","last_clean":"2024-04-18T10:06:00.384238+0000","last_became_active":"2024-04-18T10:05:59.236530+0000","last_became_peered":"2024-04-18T10:05:59.236530+0000","last_unstale":"2024-04-18T10:06:00.384238+0000","last_undegraded":"2024-04-18T10:06:00.384238+0000","last_fullsized":"2024-04-18T10:06:00.384238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:51.419833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384926+0000","last_change":"2024-04-18T10:05:59.243032+0000","last_active":"2024-04-18T10:06:00.384926+0000","last_peered":"2024-04-18T10:06:00.384926+0000","last_clean":"2024-04-18T10:06:00.384926+0000","last_became_active":"2024-04-18T10:05:59.242716+0000","last_became_peered":"2024-04-18T10:05:59.242716+0000","last_unstale":"2024-04-18T10:06:00.384926+0000","last_undegraded":"2024-04-18T10:06:00.384926+0000","last_fullsized":"2024-04-18T10:06:00.384926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384052+0000","last_change":"2024-04-18T10:05:59.240754+0000","last_active":"2024-04-18T10:06:00.384052+0000","last_peered":"2024-04-18T10:06:00.384052+0000","last_clean":"2024-04-18T10:06:00.384052+0000","last_became_active":"2024-04-18T10:05:59.239992+0000","last_became_peered":"2024-04-18T10:05:59.239992+0000","last_unstale":"2024-04-18T10:06:00.384052+0000","last_undegraded":"2024-04-18T10:06:00.384052+0000","last_fullsized":"2024-04-18T10:06:00.384052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232972+0000","last_change":"2024-04-18T10:05:59.224563+0000","last_active":"2024-04-18T10:06:00.232972+0000","last_peered":"2024-04-18T10:06:00.232972+0000","last_clean":"2024-04-18T10:06:00.232972+0000","last_became_active":"2024-04-18T10:05:59.224115+0000","last_became_peered":"2024-04-18T10:05:59.224115+0000","last_unstale":"2024-04-18T10:06:00.232972+0000","last_undegraded":"2024-04-18T10:06:00.232972+0000","last_fullsized":"2024-04-18T10:06:00.232972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238461+0000","last_change":"2024-04-18T10:05:59.232777+0000","last_active":"2024-04-18T10:06:00.238461+0000","last_peered":"2024-04-18T10:06:00.238461+0000","last_clean":"2024-04-18T10:06:00.238461+0000","last_became_active":"2024-04-18T10:05:59.231188+0000","last_became_peered":"2024-04-18T10:05:59.231188+0000","last_unstale":"2024-04-18T10:06:00.238461+0000","last_undegraded":"2024-04-18T10:06:00.238461+0000","last_fullsized":"2024-04-18T10:06:00.238461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243840+0000","last_change":"2024-04-18T10:05:59.264529+0000","last_active":"2024-04-18T10:06:00.243840+0000","last_peered":"2024-04-18T10:06:00.243840+0000","last_clean":"2024-04-18T10:06:00.243840+0000","last_became_active":"2024-04-18T10:05:59.260972+0000","last_became_peered":"2024-04-18T10:05:59.260972+0000","last_unstale":"2024-04-18T10:06:00.243840+0000","last_undegraded":"2024-04-18T10:06:00.243840+0000","last_fullsized":"2024-04-18T10:06:00.243840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:24:15.929261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385847+0000","last_change":"2024-04-18T10:05:59.262102+0000","last_active":"2024-04-18T10:06:00.385847+0000","last_peered":"2024-04-18T10:06:00.385847+0000","last_clean":"2024-04-18T10:06:00.385847+0000","last_became_active":"2024-04-18T10:05:59.261796+0000","last_became_peered":"2024-04-18T10:05:59.261796+0000","last_unstale":"2024-04-18T10:06:00.385847+0000","last_undegraded":"2024-04-18T10:06:00.385847+0000","last_fullsized":"2024-04-18T10:06:00.385847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:51.419833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241173+0000","last_change":"2024-04-18T10:05:59.256981+0000","last_active":"2024-04-18T10:06:00.241173+0000","last_peered":"2024-04-18T10:06:00.241173+0000","last_clean":"2024-04-18T10:06:00.241173+0000","last_became_active":"2024-04-18T10:05:59.252309+0000","last_became_peered":"2024-04-18T10:05:59.252309+0000","last_unstale":"2024-04-18T10:06:00.241173+0000","last_undegraded":"2024-04-18T10:06:00.241173+0000","last_fullsized":"2024-04-18T10:06:00.241173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:16:23.709955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243343+0000","last_change":"2024-04-18T10:05:59.266020+0000","last_active":"2024-04-18T10:06:00.243343+0000","last_peered":"2024-04-18T10:06:00.243343+0000","last_clean":"2024-04-18T10:06:00.243343+0000","last_became_active":"2024-04-18T10:05:59.265361+0000","last_became_peered":"2024-04-18T10:05:59.265361+0000","last_unstale":"2024-04-18T10:06:00.243343+0000","last_undegraded":"2024-04-18T10:06:00.243343+0000","last_fullsized":"2024-04-18T10:06:00.243343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384730+0000","last_change":"2024-04-18T10:05:59.262190+0000","last_active":"2024-04-18T10:06:00.384730+0000","last_peered":"2024-04-18T10:06:00.384730+0000","last_clean":"2024-04-18T10:06:00.384730+0000","last_became_active":"2024-04-18T10:05:59.261866+0000","last_became_peered":"2024-04-18T10:05:59.261866+0000","last_unstale":"2024-04-18T10:06:00.384730+0000","last_undegraded":"2024-04-18T10:06:00.384730+0000","last_fullsized":"2024-04-18T10:06:00.384730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:49:34.764977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238819+0000","last_change":"2024-04-18T10:05:59.241285+0000","last_active":"2024-04-18T10:06:00.238819+0000","last_peered":"2024-04-18T10:06:00.238819+0000","last_clean":"2024-04-18T10:06:00.238819+0000","last_became_active":"2024-04-18T10:05:59.239568+0000","last_became_peered":"2024-04-18T10:05:59.239568+0000","last_unstale":"2024-04-18T10:06:00.238819+0000","last_undegraded":"2024-04-18T10:06:00.238819+0000","last_fullsized":"2024-04-18T10:06:00.238819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:12:17.167339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385177+0000","last_change":"2024-04-18T10:05:59.264180+0000","last_active":"2024-04-18T10:06:00.385177+0000","last_peered":"2024-04-18T10:06:00.385177+0000","last_clean":"2024-04-18T10:06:00.385177+0000","last_became_active":"2024-04-18T10:05:59.263866+0000","last_became_peered":"2024-04-18T10:05:59.263866+0000","last_unstale":"2024-04-18T10:06:00.385177+0000","last_undegraded":"2024-04-18T10:06:00.385177+0000","last_fullsized":"2024-04-18T10:06:00.385177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238829+0000","last_change":"2024-04-18T10:05:59.246355+0000","last_active":"2024-04-18T10:06:00.238829+0000","last_peered":"2024-04-18T10:06:00.238829+0000","last_clean":"2024-04-18T10:06:00.238829+0000","last_became_active":"2024-04-18T10:05:59.239576+0000","last_became_peered":"2024-04-18T10:05:59.239576+0000","last_unstale":"2024-04-18T10:06:00.238829+0000","last_undegraded":"2024-04-18T10:06:00.238829+0000","last_fullsized":"2024-04-18T10:06:00.238829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:27:56.094004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386366+0000","last_change":"2024-04-18T10:05:59.264586+0000","last_active":"2024-04-18T10:06:00.386366+0000","last_peered":"2024-04-18T10:06:00.386366+0000","last_clean":"2024-04-18T10:06:00.386366+0000","last_became_active":"2024-04-18T10:05:59.264215+0000","last_became_peered":"2024-04-18T10:05:59.264215+0000","last_unstale":"2024-04-18T10:06:00.386366+0000","last_undegraded":"2024-04-18T10:06:00.386366+0000","last_fullsized":"2024-04-18T10:06:00.386366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:27:56.094004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242998+0000","last_change":"2024-04-18T10:05:59.243781+0000","last_active":"2024-04-18T10:06:00.242998+0000","last_peered":"2024-04-18T10:06:00.242998+0000","last_clean":"2024-04-18T10:06:00.242998+0000","last_became_active":"2024-04-18T10:05:59.243092+0000","last_became_peered":"2024-04-18T10:05:59.243092+0000","last_unstale":"2024-04-18T10:06:00.242998+0000","last_undegraded":"2024-04-18T10:06:00.242998+0000","last_fullsized":"2024-04-18T10:06:00.242998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:49:34.764977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232294+0000","last_change":"2024-04-18T10:05:59.234128+0000","last_active":"2024-04-18T10:06:00.232294+0000","last_peered":"2024-04-18T10:06:00.232294+0000","last_clean":"2024-04-18T10:06:00.232294+0000","last_became_active":"2024-04-18T10:05:59.233742+0000","last_became_peered":"2024-04-18T10:05:59.233742+0000","last_unstale":"2024-04-18T10:06:00.232294+0000","last_undegraded":"2024-04-18T10:06:00.232294+0000","last_fullsized":"2024-04-18T10:06:00.232294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386224+0000","last_change":"2024-04-18T10:05:59.264826+0000","last_active":"2024-04-18T10:06:00.386224+0000","last_peered":"2024-04-18T10:06:00.386224+0000","last_clean":"2024-04-18T10:06:00.386224+0000","last_became_active":"2024-04-18T10:05:59.263655+0000","last_became_peered":"2024-04-18T10:05:59.263655+0000","last_unstale":"2024-04-18T10:06:00.386224+0000","last_undegraded":"2024-04-18T10:06:00.386224+0000","last_fullsized":"2024-04-18T10:06:00.386224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:12:17.167339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383983+0000","last_change":"2024-04-18T10:05:59.235032+0000","last_active":"2024-04-18T10:06:00.383983+0000","last_peered":"2024-04-18T10:06:00.383983+0000","last_clean":"2024-04-18T10:06:00.383983+0000","last_became_active":"2024-04-18T10:05:59.234660+0000","last_became_peered":"2024-04-18T10:05:59.234660+0000","last_unstale":"2024-04-18T10:06:00.383983+0000","last_undegraded":"2024-04-18T10:06:00.383983+0000","last_fullsized":"2024-04-18T10:06:00.383983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239182+0000","last_change":"2024-04-18T10:05:59.241915+0000","last_active":"2024-04-18T10:06:00.239182+0000","last_peered":"2024-04-18T10:06:00.239182+0000","last_clean":"2024-04-18T10:06:00.239182+0000","last_became_active":"2024-04-18T10:05:59.240519+0000","last_became_peered":"2024-04-18T10:05:59.240519+0000","last_unstale":"2024-04-18T10:06:00.239182+0000","last_undegraded":"2024-04-18T10:06:00.239182+0000","last_fullsized":"2024-04-18T10:06:00.239182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:34:28.871228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385003+0000","last_change":"2024-04-18T10:05:59.246146+0000","last_active":"2024-04-18T10:06:00.385003+0000","last_peered":"2024-04-18T10:06:00.385003+0000","last_clean":"2024-04-18T10:06:00.385003+0000","last_became_active":"2024-04-18T10:05:59.245800+0000","last_became_peered":"2024-04-18T10:05:59.245800+0000","last_unstale":"2024-04-18T10:06:00.385003+0000","last_undegraded":"2024-04-18T10:06:00.385003+0000","last_fullsized":"2024-04-18T10:06:00.385003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:22.118304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242883+0000","last_change":"2024-04-18T10:05:59.239684+0000","last_active":"2024-04-18T10:06:00.242883+0000","last_peered":"2024-04-18T10:06:00.242883+0000","last_clean":"2024-04-18T10:06:00.242883+0000","last_became_active":"2024-04-18T10:05:59.233494+0000","last_became_peered":"2024-04-18T10:05:59.233494+0000","last_unstale":"2024-04-18T10:06:00.242883+0000","last_undegraded":"2024-04-18T10:06:00.242883+0000","last_fullsized":"2024-04-18T10:06:00.242883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:23:54.569394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383451+0000","last_change":"2024-04-18T10:05:59.219492+0000","last_active":"2024-04-18T10:06:00.383451+0000","last_peered":"2024-04-18T10:06:00.383451+0000","last_clean":"2024-04-18T10:06:00.383451+0000","last_became_active":"2024-04-18T10:05:59.218737+0000","last_became_peered":"2024-04-18T10:05:59.218737+0000","last_unstale":"2024-04-18T10:06:00.383451+0000","last_undegraded":"2024-04-18T10:06:00.383451+0000","last_fullsized":"2024-04-18T10:06:00.383451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:33.010147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386355+0000","last_change":"2024-04-18T10:05:58.253056+0000","last_active":"2024-04-18T10:06:00.386355+0000","last_peered":"2024-04-18T10:06:00.386355+0000","last_clean":"2024-04-18T10:06:00.386355+0000","last_became_active":"2024-04-18T10:05:58.252737+0000","last_became_peered":"2024-04-18T10:05:58.252737+0000","last_unstale":"2024-04-18T10:06:00.386355+0000","last_undegraded":"2024-04-18T10:06:00.386355+0000","last_fullsized":"2024-04-18T10:06:00.386355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384430+0000","last_change":"2024-04-18T10:05:59.263750+0000","last_active":"2024-04-18T10:06:00.384430+0000","last_peered":"2024-04-18T10:06:00.384430+0000","last_clean":"2024-04-18T10:06:00.384430+0000","last_became_active":"2024-04-18T10:05:59.263538+0000","last_became_peered":"2024-04-18T10:05:59.263538+0000","last_unstale":"2024-04-18T10:06:00.384430+0000","last_undegraded":"2024-04-18T10:06:00.384430+0000","last_fullsized":"2024-04-18T10:06:00.384430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243880+0000","last_change":"2024-04-18T10:05:58.243026+0000","last_active":"2024-04-18T10:06:00.243880+0000","last_peered":"2024-04-18T10:06:00.243880+0000","last_clean":"2024-04-18T10:06:00.243880+0000","last_became_active":"2024-04-18T10:05:58.240506+0000","last_became_peered":"2024-04-18T10:05:58.240506+0000","last_unstale":"2024-04-18T10:06:00.243880+0000","last_undegraded":"2024-04-18T10:06:00.243880+0000","last_fullsized":"2024-04-18T10:06:00.243880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385279+0000","last_change":"2024-04-18T10:05:58.243128+0000","last_active":"2024-04-18T10:06:00.385279+0000","last_peered":"2024-04-18T10:06:00.385279+0000","last_clean":"2024-04-18T10:06:00.385279+0000","last_became_active":"2024-04-18T10:05:58.242694+0000","last_became_peered":"2024-04-18T10:05:58.242694+0000","last_unstale":"2024-04-18T10:06:00.385279+0000","last_undegraded":"2024-04-18T10:06:00.385279+0000","last_fullsized":"2024-04-18T10:06:00.385279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:52:26.587547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383890+0000","last_change":"2024-04-18T10:05:59.232704+0000","last_active":"2024-04-18T10:06:00.383890+0000","last_peered":"2024-04-18T10:06:00.383890+0000","last_clean":"2024-04-18T10:06:00.383890+0000","last_became_active":"2024-04-18T10:05:59.231762+0000","last_became_peered":"2024-04-18T10:05:59.231762+0000","last_unstale":"2024-04-18T10:06:00.383890+0000","last_undegraded":"2024-04-18T10:06:00.383890+0000","last_fullsized":"2024-04-18T10:06:00.383890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239055+0000","last_change":"2024-04-18T10:05:58.231745+0000","last_active":"2024-04-18T10:06:00.239055+0000","last_peered":"2024-04-18T10:06:00.239055+0000","last_clean":"2024-04-18T10:06:00.239055+0000","last_became_active":"2024-04-18T10:05:58.230994+0000","last_became_peered":"2024-04-18T10:05:58.230994+0000","last_unstale":"2024-04-18T10:06:00.239055+0000","last_undegraded":"2024-04-18T10:06:00.239055+0000","last_fullsized":"2024-04-18T10:06:00.239055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232217+0000","last_change":"2024-04-18T10:05:59.232550+0000","last_active":"2024-04-18T10:06:00.232217+0000","last_peered":"2024-04-18T10:06:00.232217+0000","last_clean":"2024-04-18T10:06:00.232217+0000","last_became_active":"2024-04-18T10:05:59.231955+0000","last_became_peered":"2024-04-18T10:05:59.231955+0000","last_unstale":"2024-04-18T10:06:00.232217+0000","last_undegraded":"2024-04-18T10:06:00.232217+0000","last_fullsized":"2024-04-18T10:06:00.232217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243516+0000","last_change":"2024-04-18T10:05:58.244903+0000","last_active":"2024-04-18T10:06:00.243516+0000","last_peered":"2024-04-18T10:06:00.243516+0000","last_clean":"2024-04-18T10:06:00.243516+0000","last_became_active":"2024-04-18T10:05:58.244287+0000","last_became_peered":"2024-04-18T10:05:58.244287+0000","last_unstale":"2024-04-18T10:06:00.243516+0000","last_undegraded":"2024-04-18T10:06:00.243516+0000","last_fullsized":"2024-04-18T10:06:00.243516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240811+0000","last_change":"2024-04-18T10:05:59.238308+0000","last_active":"2024-04-18T10:06:00.240811+0000","last_peered":"2024-04-18T10:06:00.240811+0000","last_clean":"2024-04-18T10:06:00.240811+0000","last_became_active":"2024-04-18T10:05:59.237422+0000","last_became_peered":"2024-04-18T10:05:59.237422+0000","last_unstale":"2024-04-18T10:06:00.240811+0000","last_undegraded":"2024-04-18T10:06:00.240811+0000","last_fullsized":"2024-04-18T10:06:00.240811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:28:59.797065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240802+0000","last_change":"2024-04-18T10:05:58.245925+0000","last_active":"2024-04-18T10:06:00.240802+0000","last_peered":"2024-04-18T10:06:00.240802+0000","last_clean":"2024-04-18T10:06:00.240802+0000","last_became_active":"2024-04-18T10:05:58.244713+0000","last_became_peered":"2024-04-18T10:05:58.244713+0000","last_unstale":"2024-04-18T10:06:00.240802+0000","last_undegraded":"2024-04-18T10:06:00.240802+0000","last_fullsized":"2024-04-18T10:06:00.240802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384230+0000","last_change":"2024-04-18T10:05:59.252961+0000","last_active":"2024-04-18T10:06:00.384230+0000","last_peered":"2024-04-18T10:06:00.384230+0000","last_clean":"2024-04-18T10:06:00.384230+0000","last_became_active":"2024-04-18T10:05:59.252672+0000","last_became_peered":"2024-04-18T10:05:59.252672+0000","last_unstale":"2024-04-18T10:06:00.384230+0000","last_undegraded":"2024-04-18T10:06:00.384230+0000","last_fullsized":"2024-04-18T10:06:00.384230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232406+0000","last_change":"2024-04-18T10:05:58.235997+0000","last_active":"2024-04-18T10:06:00.232406+0000","last_peered":"2024-04-18T10:06:00.232406+0000","last_clean":"2024-04-18T10:06:00.232406+0000","last_became_active":"2024-04-18T10:05:58.235330+0000","last_became_peered":"2024-04-18T10:05:58.235330+0000","last_unstale":"2024-04-18T10:06:00.232406+0000","last_undegraded":"2024-04-18T10:06:00.232406+0000","last_fullsized":"2024-04-18T10:06:00.232406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:21.107270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384433+0000","last_change":"2024-04-18T10:05:59.257176+0000","last_active":"2024-04-18T10:06:00.384433+0000","last_peered":"2024-04-18T10:06:00.384433+0000","last_clean":"2024-04-18T10:06:00.384433+0000","last_became_active":"2024-04-18T10:05:59.256452+0000","last_became_peered":"2024-04-18T10:05:59.256452+0000","last_unstale":"2024-04-18T10:06:00.384433+0000","last_undegraded":"2024-04-18T10:06:00.384433+0000","last_fullsized":"2024-04-18T10:06:00.384433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:04:27.662085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243463+0000","last_change":"2024-04-18T10:05:58.239207+0000","last_active":"2024-04-18T10:06:00.243463+0000","last_peered":"2024-04-18T10:06:00.243463+0000","last_clean":"2024-04-18T10:06:00.243463+0000","last_became_active":"2024-04-18T10:05:58.238349+0000","last_became_peered":"2024-04-18T10:05:58.238349+0000","last_unstale":"2024-04-18T10:06:00.243463+0000","last_undegraded":"2024-04-18T10:06:00.243463+0000","last_fullsized":"2024-04-18T10:06:00.243463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239372+0000","last_change":"2024-04-18T10:05:59.254087+0000","last_active":"2024-04-18T10:06:00.239372+0000","last_peered":"2024-04-18T10:06:00.239372+0000","last_clean":"2024-04-18T10:06:00.239372+0000","last_became_active":"2024-04-18T10:05:59.253296+0000","last_became_peered":"2024-04-18T10:05:59.253296+0000","last_unstale":"2024-04-18T10:06:00.239372+0000","last_undegraded":"2024-04-18T10:06:00.239372+0000","last_fullsized":"2024-04-18T10:06:00.239372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:27:56.094004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384701+0000","last_change":"2024-04-18T10:05:58.236375+0000","last_active":"2024-04-18T10:06:00.384701+0000","last_peered":"2024-04-18T10:06:00.384701+0000","last_clean":"2024-04-18T10:06:00.384701+0000","last_became_active":"2024-04-18T10:05:58.236066+0000","last_became_peered":"2024-04-18T10:05:58.236066+0000","last_unstale":"2024-04-18T10:06:00.384701+0000","last_undegraded":"2024-04-18T10:06:00.384701+0000","last_fullsized":"2024-04-18T10:06:00.384701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:52:26.587547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383621+0000","last_change":"2024-04-18T10:05:59.262667+0000","last_active":"2024-04-18T10:06:00.383621+0000","last_peered":"2024-04-18T10:06:00.383621+0000","last_clean":"2024-04-18T10:06:00.383621+0000","last_became_active":"2024-04-18T10:05:59.262365+0000","last_became_peered":"2024-04-18T10:05:59.262365+0000","last_unstale":"2024-04-18T10:06:00.383621+0000","last_undegraded":"2024-04-18T10:06:00.383621+0000","last_fullsized":"2024-04-18T10:06:00.383621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:23:54.569394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239431+0000","last_change":"2024-04-18T10:05:58.243114+0000","last_active":"2024-04-18T10:06:00.239431+0000","last_peered":"2024-04-18T10:06:00.239431+0000","last_clean":"2024-04-18T10:06:00.239431+0000","last_became_active":"2024-04-18T10:05:58.242102+0000","last_became_peered":"2024-04-18T10:05:58.242102+0000","last_unstale":"2024-04-18T10:06:00.239431+0000","last_undegraded":"2024-04-18T10:06:00.239431+0000","last_fullsized":"2024-04-18T10:06:00.239431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239878+0000","last_change":"2024-04-18T10:05:59.274245+0000","last_active":"2024-04-18T10:06:00.239878+0000","last_peered":"2024-04-18T10:06:00.239878+0000","last_clean":"2024-04-18T10:06:00.239878+0000","last_became_active":"2024-04-18T10:05:59.273707+0000","last_became_peered":"2024-04-18T10:05:59.273707+0000","last_unstale":"2024-04-18T10:06:00.239878+0000","last_undegraded":"2024-04-18T10:06:00.239878+0000","last_fullsized":"2024-04-18T10:06:00.239878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:49.354229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243436+0000","last_change":"2024-04-18T10:05:58.242951+0000","last_active":"2024-04-18T10:06:00.243436+0000","last_peered":"2024-04-18T10:06:00.243436+0000","last_clean":"2024-04-18T10:06:00.243436+0000","last_became_active":"2024-04-18T10:05:58.241438+0000","last_became_peered":"2024-04-18T10:05:58.241438+0000","last_unstale":"2024-04-18T10:06:00.243436+0000","last_undegraded":"2024-04-18T10:06:00.243436+0000","last_fullsized":"2024-04-18T10:06:00.243436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243154+0000","last_change":"2024-04-18T10:05:59.233588+0000","last_active":"2024-04-18T10:06:00.243154+0000","last_peered":"2024-04-18T10:06:00.243154+0000","last_clean":"2024-04-18T10:06:00.243154+0000","last_became_active":"2024-04-18T10:05:59.232831+0000","last_became_peered":"2024-04-18T10:05:59.232831+0000","last_unstale":"2024-04-18T10:06:00.243154+0000","last_undegraded":"2024-04-18T10:06:00.243154+0000","last_fullsized":"2024-04-18T10:06:00.243154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384620+0000","last_change":"2024-04-18T10:05:58.223796+0000","last_active":"2024-04-18T10:06:00.384620+0000","last_peered":"2024-04-18T10:06:00.384620+0000","last_clean":"2024-04-18T10:06:00.384620+0000","last_became_active":"2024-04-18T10:05:58.223248+0000","last_became_peered":"2024-04-18T10:05:58.223248+0000","last_unstale":"2024-04-18T10:06:00.384620+0000","last_undegraded":"2024-04-18T10:06:00.384620+0000","last_fullsized":"2024-04-18T10:06:00.384620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:52:26.587547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240028+0000","last_change":"2024-04-18T10:05:59.277176+0000","last_active":"2024-04-18T10:06:00.240028+0000","last_peered":"2024-04-18T10:06:00.240028+0000","last_clean":"2024-04-18T10:06:00.240028+0000","last_became_active":"2024-04-18T10:05:59.276386+0000","last_became_peered":"2024-04-18T10:05:59.276386+0000","last_unstale":"2024-04-18T10:06:00.240028+0000","last_undegraded":"2024-04-18T10:06:00.240028+0000","last_fullsized":"2024-04-18T10:06:00.240028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:28:59.797065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384638+0000","last_change":"2024-04-18T10:05:58.233338+0000","last_active":"2024-04-18T10:06:00.384638+0000","last_peered":"2024-04-18T10:06:00.384638+0000","last_clean":"2024-04-18T10:06:00.384638+0000","last_became_active":"2024-04-18T10:05:58.231906+0000","last_became_peered":"2024-04-18T10:05:58.231906+0000","last_unstale":"2024-04-18T10:06:00.384638+0000","last_undegraded":"2024-04-18T10:06:00.384638+0000","last_fullsized":"2024-04-18T10:06:00.384638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:06:57.988393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241935+0000","last_change":"2024-04-18T10:05:59.275977+0000","last_active":"2024-04-18T10:06:00.241935+0000","last_peered":"2024-04-18T10:06:00.241935+0000","last_clean":"2024-04-18T10:06:00.241935+0000","last_became_active":"2024-04-18T10:05:59.275386+0000","last_became_peered":"2024-04-18T10:05:59.275386+0000","last_unstale":"2024-04-18T10:06:00.241935+0000","last_undegraded":"2024-04-18T10:06:00.241935+0000","last_fullsized":"2024-04-18T10:06:00.241935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:58:16.367066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386132+0000","last_change":"2024-04-18T10:05:58.229897+0000","last_active":"2024-04-18T10:06:00.386132+0000","last_peered":"2024-04-18T10:06:00.386132+0000","last_clean":"2024-04-18T10:06:00.386132+0000","last_became_active":"2024-04-18T10:05:58.229451+0000","last_became_peered":"2024-04-18T10:05:58.229451+0000","last_unstale":"2024-04-18T10:06:00.386132+0000","last_undegraded":"2024-04-18T10:06:00.386132+0000","last_fullsized":"2024-04-18T10:06:00.386132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242763+0000","last_change":"2024-04-18T10:05:59.276815+0000","last_active":"2024-04-18T10:06:00.242763+0000","last_peered":"2024-04-18T10:06:00.242763+0000","last_clean":"2024-04-18T10:06:00.242763+0000","last_became_active":"2024-04-18T10:05:59.276272+0000","last_became_peered":"2024-04-18T10:05:59.276272+0000","last_unstale":"2024-04-18T10:06:00.242763+0000","last_undegraded":"2024-04-18T10:06:00.242763+0000","last_fullsized":"2024-04-18T10:06:00.242763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:44:45.775310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232479+0000","last_change":"2024-04-18T10:05:58.230942+0000","last_active":"2024-04-18T10:06:00.232479+0000","last_peered":"2024-04-18T10:06:00.232479+0000","last_clean":"2024-04-18T10:06:00.232479+0000","last_became_active":"2024-04-18T10:05:58.230373+0000","last_became_peered":"2024-04-18T10:05:58.230373+0000","last_unstale":"2024-04-18T10:06:00.232479+0000","last_undegraded":"2024-04-18T10:06:00.232479+0000","last_fullsized":"2024-04-18T10:06:00.232479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:50.408798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241359+0000","last_change":"2024-04-18T10:05:59.234199+0000","last_active":"2024-04-18T10:06:00.241359+0000","last_peered":"2024-04-18T10:06:00.241359+0000","last_clean":"2024-04-18T10:06:00.241359+0000","last_became_active":"2024-04-18T10:05:59.233412+0000","last_became_peered":"2024-04-18T10:05:59.233412+0000","last_unstale":"2024-04-18T10:06:00.241359+0000","last_undegraded":"2024-04-18T10:06:00.241359+0000","last_fullsized":"2024-04-18T10:06:00.241359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:16:23.709955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385242+0000","last_change":"2024-04-18T10:05:58.222111+0000","last_active":"2024-04-18T10:06:00.385242+0000","last_peered":"2024-04-18T10:06:00.385242+0000","last_clean":"2024-04-18T10:06:00.385242+0000","last_became_active":"2024-04-18T10:05:58.221783+0000","last_became_peered":"2024-04-18T10:05:58.221783+0000","last_unstale":"2024-04-18T10:06:00.385242+0000","last_undegraded":"2024-04-18T10:06:00.385242+0000","last_fullsized":"2024-04-18T10:06:00.385242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240501+0000","last_change":"2024-04-18T10:05:59.261884+0000","last_active":"2024-04-18T10:06:00.240501+0000","last_peered":"2024-04-18T10:06:00.240501+0000","last_clean":"2024-04-18T10:06:00.240501+0000","last_became_active":"2024-04-18T10:05:59.261147+0000","last_became_peered":"2024-04-18T10:05:59.261147+0000","last_unstale":"2024-04-18T10:06:00.240501+0000","last_undegraded":"2024-04-18T10:06:00.240501+0000","last_fullsized":"2024-04-18T10:06:00.240501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:44:42.861177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238296+0000","last_change":"2024-04-18T10:05:49.183850+0000","last_active":"2024-04-18T10:06:00.238296+0000","last_peered":"2024-04-18T10:06:00.238296+0000","last_clean":"2024-04-18T10:06:00.238296+0000","last_became_active":"2024-04-18T10:05:49.181471+0000","last_became_peered":"2024-04-18T10:05:49.181471+0000","last_unstale":"2024-04-18T10:06:00.238296+0000","last_undegraded":"2024-04-18T10:06:00.238296+0000","last_fullsized":"2024-04-18T10:06:00.238296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_clean_scrub_stamp":"2024-04-18T10:05:47.511486+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:06:51.214547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239978+0000","last_change":"2024-04-18T10:05:58.228257+0000","last_active":"2024-04-18T10:06:00.239978+0000","last_peered":"2024-04-18T10:06:00.239978+0000","last_clean":"2024-04-18T10:06:00.239978+0000","last_became_active":"2024-04-18T10:05:58.224594+0000","last_became_peered":"2024-04-18T10:05:58.224594+0000","last_unstale":"2024-04-18T10:06:00.239978+0000","last_undegraded":"2024-04-18T10:06:00.239978+0000","last_fullsized":"2024-04-18T10:06:00.239978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385522+0000","last_change":"2024-04-18T10:05:59.256229+0000","last_active":"2024-04-18T10:06:00.385522+0000","last_peered":"2024-04-18T10:06:00.385522+0000","last_clean":"2024-04-18T10:06:00.385522+0000","last_became_active":"2024-04-18T10:05:59.255894+0000","last_became_peered":"2024-04-18T10:05:59.255894+0000","last_unstale":"2024-04-18T10:06:00.385522+0000","last_undegraded":"2024-04-18T10:06:00.385522+0000","last_fullsized":"2024-04-18T10:06:00.385522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238224+0000","last_change":"2024-04-18T10:05:58.254200+0000","last_active":"2024-04-18T10:06:00.238224+0000","last_peered":"2024-04-18T10:06:00.238224+0000","last_clean":"2024-04-18T10:06:00.238224+0000","last_became_active":"2024-04-18T10:05:58.253365+0000","last_became_peered":"2024-04-18T10:05:58.253365+0000","last_unstale":"2024-04-18T10:06:00.238224+0000","last_undegraded":"2024-04-18T10:06:00.238224+0000","last_fullsized":"2024-04-18T10:06:00.238224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383217+0000","last_change":"2024-04-18T10:05:59.264229+0000","last_active":"2024-04-18T10:06:00.383217+0000","last_peered":"2024-04-18T10:06:00.383217+0000","last_clean":"2024-04-18T10:06:00.383217+0000","last_became_active":"2024-04-18T10:05:59.263577+0000","last_became_peered":"2024-04-18T10:05:59.263577+0000","last_unstale":"2024-04-18T10:06:00.383217+0000","last_undegraded":"2024-04-18T10:06:00.383217+0000","last_fullsized":"2024-04-18T10:06:00.383217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:37:20.126666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385257+0000","last_change":"2024-04-18T10:05:58.235466+0000","last_active":"2024-04-18T10:06:00.385257+0000","last_peered":"2024-04-18T10:06:00.385257+0000","last_clean":"2024-04-18T10:06:00.385257+0000","last_became_active":"2024-04-18T10:05:58.234823+0000","last_became_peered":"2024-04-18T10:05:58.234823+0000","last_unstale":"2024-04-18T10:06:00.385257+0000","last_undegraded":"2024-04-18T10:06:00.385257+0000","last_fullsized":"2024-04-18T10:06:00.385257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232451+0000","last_change":"2024-04-18T10:05:59.230409+0000","last_active":"2024-04-18T10:06:00.232451+0000","last_peered":"2024-04-18T10:06:00.232451+0000","last_clean":"2024-04-18T10:06:00.232451+0000","last_became_active":"2024-04-18T10:05:59.228899+0000","last_became_peered":"2024-04-18T10:05:59.228899+0000","last_unstale":"2024-04-18T10:06:00.232451+0000","last_undegraded":"2024-04-18T10:06:00.232451+0000","last_fullsized":"2024-04-18T10:06:00.232451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239789+0000","last_change":"2024-04-18T10:05:58.232028+0000","last_active":"2024-04-18T10:06:00.239789+0000","last_peered":"2024-04-18T10:06:00.239789+0000","last_clean":"2024-04-18T10:06:00.239789+0000","last_became_active":"2024-04-18T10:05:58.229311+0000","last_became_peered":"2024-04-18T10:05:58.229311+0000","last_unstale":"2024-04-18T10:06:00.239789+0000","last_undegraded":"2024-04-18T10:06:00.239789+0000","last_fullsized":"2024-04-18T10:06:00.239789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383872+0000","last_change":"2024-04-18T10:05:59.248629+0000","last_active":"2024-04-18T10:06:00.383872+0000","last_peered":"2024-04-18T10:06:00.383872+0000","last_clean":"2024-04-18T10:06:00.383872+0000","last_became_active":"2024-04-18T10:05:59.247619+0000","last_became_peered":"2024-04-18T10:05:59.247619+0000","last_unstale":"2024-04-18T10:06:00.383872+0000","last_undegraded":"2024-04-18T10:06:00.383872+0000","last_fullsized":"2024-04-18T10:06:00.383872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:35.965832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232588+0000","last_change":"2024-04-18T10:05:58.224570+0000","last_active":"2024-04-18T10:06:00.232588+0000","last_peered":"2024-04-18T10:06:00.232588+0000","last_clean":"2024-04-18T10:06:00.232588+0000","last_became_active":"2024-04-18T10:05:58.224216+0000","last_became_peered":"2024-04-18T10:05:58.224216+0000","last_unstale":"2024-04-18T10:06:00.232588+0000","last_undegraded":"2024-04-18T10:06:00.232588+0000","last_fullsized":"2024-04-18T10:06:00.232588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242688+0000","last_change":"2024-04-18T10:05:59.246784+0000","last_active":"2024-04-18T10:06:00.242688+0000","last_peered":"2024-04-18T10:06:00.242688+0000","last_clean":"2024-04-18T10:06:00.242688+0000","last_became_active":"2024-04-18T10:05:59.243518+0000","last_became_peered":"2024-04-18T10:05:59.243518+0000","last_unstale":"2024-04-18T10:06:00.242688+0000","last_undegraded":"2024-04-18T10:06:00.242688+0000","last_fullsized":"2024-04-18T10:06:00.242688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:45:43.670227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241271+0000","last_change":"2024-04-18T10:05:58.258428+0000","last_active":"2024-04-18T10:06:00.241271+0000","last_peered":"2024-04-18T10:06:00.241271+0000","last_clean":"2024-04-18T10:06:00.241271+0000","last_became_active":"2024-04-18T10:05:58.257648+0000","last_became_peered":"2024-04-18T10:05:58.257648+0000","last_unstale":"2024-04-18T10:06:00.241271+0000","last_undegraded":"2024-04-18T10:06:00.241271+0000","last_fullsized":"2024-04-18T10:06:00.241271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:15:05.853314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232543+0000","last_change":"2024-04-18T10:05:59.229368+0000","last_active":"2024-04-18T10:06:00.232543+0000","last_peered":"2024-04-18T10:06:00.232543+0000","last_clean":"2024-04-18T10:06:00.232543+0000","last_became_active":"2024-04-18T10:05:59.228912+0000","last_became_peered":"2024-04-18T10:05:59.228912+0000","last_unstale":"2024-04-18T10:06:00.232543+0000","last_undegraded":"2024-04-18T10:06:00.232543+0000","last_fullsized":"2024-04-18T10:06:00.232543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241350+0000","last_change":"2024-04-18T10:05:58.243743+0000","last_active":"2024-04-18T10:06:00.241350+0000","last_peered":"2024-04-18T10:06:00.241350+0000","last_clean":"2024-04-18T10:06:00.241350+0000","last_became_active":"2024-04-18T10:05:58.243448+0000","last_became_peered":"2024-04-18T10:05:58.243448+0000","last_unstale":"2024-04-18T10:06:00.241350+0000","last_undegraded":"2024-04-18T10:06:00.241350+0000","last_fullsized":"2024-04-18T10:06:00.241350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:04:26.651051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386763+0000","last_change":"2024-04-18T10:05:59.262383+0000","last_active":"2024-04-18T10:06:00.386763+0000","last_peered":"2024-04-18T10:06:00.386763+0000","last_clean":"2024-04-18T10:06:00.386763+0000","last_became_active":"2024-04-18T10:05:59.262039+0000","last_became_peered":"2024-04-18T10:05:59.262039+0000","last_unstale":"2024-04-18T10:06:00.386763+0000","last_undegraded":"2024-04-18T10:06:00.386763+0000","last_fullsized":"2024-04-18T10:06:00.386763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:49.354229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243760+0000","last_change":"2024-04-18T10:05:58.251779+0000","last_active":"2024-04-18T10:06:00.243760+0000","last_peered":"2024-04-18T10:06:00.243760+0000","last_clean":"2024-04-18T10:06:00.243760+0000","last_became_active":"2024-04-18T10:05:58.251062+0000","last_became_peered":"2024-04-18T10:05:58.251062+0000","last_unstale":"2024-04-18T10:06:00.243760+0000","last_undegraded":"2024-04-18T10:06:00.243760+0000","last_fullsized":"2024-04-18T10:06:00.243760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384029+0000","last_change":"2024-04-18T10:05:59.240684+0000","last_active":"2024-04-18T10:06:00.384029+0000","last_peered":"2024-04-18T10:06:00.384029+0000","last_clean":"2024-04-18T10:06:00.384029+0000","last_became_active":"2024-04-18T10:05:59.240231+0000","last_became_peered":"2024-04-18T10:05:59.240231+0000","last_unstale":"2024-04-18T10:06:00.384029+0000","last_undegraded":"2024-04-18T10:06:00.384029+0000","last_fullsized":"2024-04-18T10:06:00.384029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238119+0000","last_change":"2024-04-18T10:05:58.254872+0000","last_active":"2024-04-18T10:06:00.238119+0000","last_peered":"2024-04-18T10:06:00.238119+0000","last_clean":"2024-04-18T10:06:00.238119+0000","last_became_active":"2024-04-18T10:05:58.252439+0000","last_became_peered":"2024-04-18T10:05:58.252439+0000","last_unstale":"2024-04-18T10:06:00.238119+0000","last_undegraded":"2024-04-18T10:06:00.238119+0000","last_fullsized":"2024-04-18T10:06:00.238119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384090+0000","last_change":"2024-04-18T10:05:59.254583+0000","last_active":"2024-04-18T10:06:00.384090+0000","last_peered":"2024-04-18T10:06:00.384090+0000","last_clean":"2024-04-18T10:06:00.384090+0000","last_became_active":"2024-04-18T10:05:59.253836+0000","last_became_peered":"2024-04-18T10:05:59.253836+0000","last_unstale":"2024-04-18T10:06:00.384090+0000","last_undegraded":"2024-04-18T10:06:00.384090+0000","last_fullsized":"2024-04-18T10:06:00.384090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:22.118304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232357+0000","last_change":"2024-04-18T10:05:58.248261+0000","last_active":"2024-04-18T10:06:00.232357+0000","last_peered":"2024-04-18T10:06:00.232357+0000","last_clean":"2024-04-18T10:06:00.232357+0000","last_became_active":"2024-04-18T10:05:58.247708+0000","last_became_peered":"2024-04-18T10:05:58.247708+0000","last_unstale":"2024-04-18T10:06:00.232357+0000","last_undegraded":"2024-04-18T10:06:00.232357+0000","last_fullsized":"2024-04-18T10:06:00.232357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384524+0000","last_change":"2024-04-18T10:05:59.238345+0000","last_active":"2024-04-18T10:06:00.384524+0000","last_peered":"2024-04-18T10:06:00.384524+0000","last_clean":"2024-04-18T10:06:00.384524+0000","last_became_active":"2024-04-18T10:05:59.237905+0000","last_became_peered":"2024-04-18T10:05:59.237905+0000","last_unstale":"2024-04-18T10:06:00.384524+0000","last_undegraded":"2024-04-18T10:06:00.384524+0000","last_fullsized":"2024-04-18T10:06:00.384524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232345+0000","last_change":"2024-04-18T10:05:58.229628+0000","last_active":"2024-04-18T10:06:00.232345+0000","last_peered":"2024-04-18T10:06:00.232345+0000","last_clean":"2024-04-18T10:06:00.232345+0000","last_became_active":"2024-04-18T10:05:58.229323+0000","last_became_peered":"2024-04-18T10:05:58.229323+0000","last_unstale":"2024-04-18T10:06:00.232345+0000","last_undegraded":"2024-04-18T10:06:00.232345+0000","last_fullsized":"2024-04-18T10:06:00.232345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385760+0000","last_change":"2024-04-18T10:05:59.232981+0000","last_active":"2024-04-18T10:06:00.385760+0000","last_peered":"2024-04-18T10:06:00.385760+0000","last_clean":"2024-04-18T10:06:00.385760+0000","last_became_active":"2024-04-18T10:05:59.232662+0000","last_became_peered":"2024-04-18T10:05:59.232662+0000","last_unstale":"2024-04-18T10:06:00.385760+0000","last_undegraded":"2024-04-18T10:06:00.385760+0000","last_fullsized":"2024-04-18T10:06:00.385760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239926+0000","last_change":"2024-04-18T10:05:58.258738+0000","last_active":"2024-04-18T10:06:00.239926+0000","last_peered":"2024-04-18T10:06:00.239926+0000","last_clean":"2024-04-18T10:06:00.239926+0000","last_became_active":"2024-04-18T10:05:58.257527+0000","last_became_peered":"2024-04-18T10:05:58.257527+0000","last_unstale":"2024-04-18T10:06:00.239926+0000","last_undegraded":"2024-04-18T10:06:00.239926+0000","last_fullsized":"2024-04-18T10:06:00.239926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:21.107270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243622+0000","last_change":"2024-04-18T10:05:59.264420+0000","last_active":"2024-04-18T10:06:00.243622+0000","last_peered":"2024-04-18T10:06:00.243622+0000","last_clean":"2024-04-18T10:06:00.243622+0000","last_became_active":"2024-04-18T10:05:59.263738+0000","last_became_peered":"2024-04-18T10:05:59.263738+0000","last_unstale":"2024-04-18T10:06:00.243622+0000","last_undegraded":"2024-04-18T10:06:00.243622+0000","last_fullsized":"2024-04-18T10:06:00.243622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242313+0000","last_change":"2024-04-18T10:05:59.222327+0000","last_active":"2024-04-18T10:06:00.242313+0000","last_peered":"2024-04-18T10:06:00.242313+0000","last_clean":"2024-04-18T10:06:00.242313+0000","last_became_active":"2024-04-18T10:05:59.221623+0000","last_became_peered":"2024-04-18T10:05:59.221623+0000","last_unstale":"2024-04-18T10:06:00.242313+0000","last_undegraded":"2024-04-18T10:06:00.242313+0000","last_fullsized":"2024-04-18T10:06:00.242313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:58:16.367066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239652+0000","last_change":"2024-04-18T10:05:58.255714+0000","last_active":"2024-04-18T10:06:00.239652+0000","last_peered":"2024-04-18T10:06:00.239652+0000","last_clean":"2024-04-18T10:06:00.239652+0000","last_became_active":"2024-04-18T10:05:58.255115+0000","last_became_peered":"2024-04-18T10:05:58.255115+0000","last_unstale":"2024-04-18T10:06:00.239652+0000","last_undegraded":"2024-04-18T10:06:00.239652+0000","last_fullsized":"2024-04-18T10:06:00.239652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:50.408798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240843+0000","last_change":"2024-04-18T10:05:59.244685+0000","last_active":"2024-04-18T10:06:00.240843+0000","last_peered":"2024-04-18T10:06:00.240843+0000","last_clean":"2024-04-18T10:06:00.240843+0000","last_became_active":"2024-04-18T10:05:59.243811+0000","last_became_peered":"2024-04-18T10:05:59.243811+0000","last_unstale":"2024-04-18T10:06:00.240843+0000","last_undegraded":"2024-04-18T10:06:00.240843+0000","last_fullsized":"2024-04-18T10:06:00.240843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:44:42.861177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241014+0000","last_change":"2024-04-18T10:05:58.256922+0000","last_active":"2024-04-18T10:06:00.241014+0000","last_peered":"2024-04-18T10:06:00.241014+0000","last_clean":"2024-04-18T10:06:00.241014+0000","last_became_active":"2024-04-18T10:05:58.255255+0000","last_became_peered":"2024-04-18T10:05:58.255255+0000","last_unstale":"2024-04-18T10:06:00.241014+0000","last_undegraded":"2024-04-18T10:06:00.241014+0000","last_fullsized":"2024-04-18T10:06:00.241014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:04:26.651051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383632+0000","last_change":"2024-04-18T10:05:59.253025+0000","last_active":"2024-04-18T10:06:00.383632+0000","last_peered":"2024-04-18T10:06:00.383632+0000","last_clean":"2024-04-18T10:06:00.383632+0000","last_became_active":"2024-04-18T10:05:59.252708+0000","last_became_peered":"2024-04-18T10:05:59.252708+0000","last_unstale":"2024-04-18T10:06:00.383632+0000","last_undegraded":"2024-04-18T10:06:00.383632+0000","last_fullsized":"2024-04-18T10:06:00.383632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:17:23.117921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386955+0000","last_change":"2024-04-18T10:05:58.234122+0000","last_active":"2024-04-18T10:06:00.386955+0000","last_peered":"2024-04-18T10:06:00.386955+0000","last_clean":"2024-04-18T10:06:00.386955+0000","last_became_active":"2024-04-18T10:05:58.233320+0000","last_became_peered":"2024-04-18T10:05:58.233320+0000","last_unstale":"2024-04-18T10:06:00.386955+0000","last_undegraded":"2024-04-18T10:06:00.386955+0000","last_fullsized":"2024-04-18T10:06:00.386955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386581+0000","last_change":"2024-04-18T10:05:59.244230+0000","last_active":"2024-04-18T10:06:00.386581+0000","last_peered":"2024-04-18T10:06:00.386581+0000","last_clean":"2024-04-18T10:06:00.386581+0000","last_became_active":"2024-04-18T10:05:59.243879+0000","last_became_peered":"2024-04-18T10:05:59.243879+0000","last_unstale":"2024-04-18T10:06:00.386581+0000","last_undegraded":"2024-04-18T10:06:00.386581+0000","last_fullsized":"2024-04-18T10:06:00.386581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:34:28.871228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383948+0000","last_change":"2024-04-18T10:05:58.247996+0000","last_active":"2024-04-18T10:06:00.383948+0000","last_peered":"2024-04-18T10:06:00.383948+0000","last_clean":"2024-04-18T10:06:00.383948+0000","last_became_active":"2024-04-18T10:05:58.247612+0000","last_became_peered":"2024-04-18T10:05:58.247612+0000","last_unstale":"2024-04-18T10:06:00.383948+0000","last_undegraded":"2024-04-18T10:06:00.383948+0000","last_fullsized":"2024-04-18T10:06:00.383948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:13:49.398609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T10:06:06.071 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T10:06:06.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.234+0000 7f0e50a5c640 1 Processor -- start 2024-04-18T10:06:06.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.234+0000 7f0e50a5c640 1 -- start start 2024-04-18T10:06:06.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e50a5c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0e4c0b9d90 0x7f0e4c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e50a5c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e4c0ba170 con 0x7f0e4c09be30 2024-04-18T10:06:06.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e50a5c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e4c14a060 con 0x7f0e4c145f20 2024-04-18T10:06:06.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e50a5c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f0e4c09cd30 con 0x7f0e4c0b9d90 2024-04-18T10:06:06.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e41d9b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0e4c09be30 0x7f0e4c06d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54984/0 (socket says 172.21.15.62:54984) 2024-04-18T10:06:06.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e41d9b640 1 -- 172.21.15.62:0/1745503298 learned_addr learned my addr 172.21.15.62:0/1745503298 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:06.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e4a59c640 1 --2- 172.21.15.62:0/1745503298 >> v2:172.21.15.145:3300/0 conn(0x7f0e4c0b9d90 0x7f0e4c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 591859755 0 0) 0x7f0e4c0ba170 con 0x7f0e4c09be30 2024-04-18T10:06:06.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e2c003530 con 0x7f0e4c09be30 2024-04-18T10:06:06.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1450596802 0 0) 0x7f0e2c003530 con 0x7f0e4c09be30 2024-04-18T10:06:06.238 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 >> v2:172.21.15.145:3300/0 conn(0x7f0e4c0b9d90 msgr2=0x7f0e4c06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 --2- 172.21.15.62:0/1745503298 >> v2:172.21.15.145:3300/0 conn(0x7f0e4c0b9d90 0x7f0e4c06d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 >> v1:172.21.15.145:6789/0 conn(0x7f0e4c145f20 legacy=0x7f0e4c1504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e4c149860 con 0x7f0e4c09be30 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f0e34001680 con 0x7f0e4c09be30 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 >> v1:172.21.15.62:6789/0 conn(0x7f0e4c09be30 legacy=0x7f0e4c06d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 --2- 172.21.15.62:0/1745503298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e2c0039c0 0x7f0e2c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 --2- 172.21.15.62:0/1745503298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e2c004810 0x7f0e2c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 --2- 172.21.15.62:0/1745503298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c0052f0 0x7f0e2c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4c09cd30 con 0x7f0e2c0052f0 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f0e2c00c1c0 con 0x7f0e2c0039c0 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f0e2c00c3c0 con 0x7f0e2c004810 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e4ad9d640 1 --2- 172.21.15.62:0/1745503298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c0052f0 0x7f0e2c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e4a59c640 1 --2- 172.21.15.62:0/1745503298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e2c0039c0 0x7f0e2c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.239 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e41d9b640 1 --2- 172.21.15.62:0/1745503298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e2c004810 0x7f0e2c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e4ad9d640 1 -- 172.21.15.62:0/1745503298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e2c004810 msgr2=0x7f0e2c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:06.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e4ad9d640 1 --2- 172.21.15.62:0/1745503298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e2c004810 0x7f0e2c004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e4ad9d640 1 -- 172.21.15.62:0/1745503298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e2c0039c0 msgr2=0x7f0e2c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:06.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e4ad9d640 1 --2- 172.21.15.62:0/1745503298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e2c0039c0 0x7f0e2c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.240 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e4ad9d640 1 -- 172.21.15.62:0/1745503298 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e2c003530 con 0x7f0e2c0052f0 2024-04-18T10:06:06.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.238+0000 7f0e4ad9d640 1 -- 172.21.15.62:0/1745503298 reap_dead start 2024-04-18T10:06:06.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e4ad9d640 1 --2- 172.21.15.62:0/1745503298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c0052f0 0x7f0e2c00b9d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0e44001e40 tx=0x7f0e4401c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a13b2597e1ff66 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:06.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e44001a90 con 0x7f0e2c0052f0 2024-04-18T10:06:06.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e44003b80 con 0x7f0e2c0052f0 2024-04-18T10:06:06.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e49d9b640 1 -- 172.21.15.62:0/1745503298 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e440050d0 con 0x7f0e2c0052f0 2024-04-18T10:06:06.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 -- 172.21.15.62:0/1745503298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c0052f0 msgr2=0x7f0e2c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 --2- 172.21.15.62:0/1745503298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c0052f0 0x7f0e2c00b9d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0e44001e40 tx=0x7f0e4401c760 comp rx=0 tx=0).stop 2024-04-18T10:06:06.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 -- 172.21.15.62:0/1745503298 shutdown_connections 2024-04-18T10:06:06.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 --2- 172.21.15.62:0/1745503298 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e2c0052f0 0x7f0e2c00b9d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 -- 172.21.15.62:0/1745503298 >> 172.21.15.62:0/1745503298 conn(0x7f0e4c06c0d0 msgr2=0x7f0e4c0b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:06.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 -- 172.21.15.62:0/1745503298 shutdown_connections 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 -- 172.21.15.62:0/1745503298 wait complete. 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 Processor -- start 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 -- start start 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e2c0052f0 0x7f0e4c13fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4c1506a0 0x7f0e4c1e3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e4c150ef0 0x7f0e4c147df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4c0b3fa0 con 0x7f0e4c1506a0 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4c0b3e20 con 0x7f0e4c150ef0 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f0e4c06d210 con 0x7f0e2c0052f0 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e41d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4c1506a0 0x7f0e4c1e3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e41d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4c1506a0 0x7f0e4c1e3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53420/0 (socket says 172.21.15.62:53420) 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e41d9b640 1 -- 172.21.15.62:0/233755204 learned_addr learned my addr 172.21.15.62:0/233755204 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e4ad9d640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e4c150ef0 0x7f0e4c147df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e4a59c640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e2c0052f0 0x7f0e4c13fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e41d9b640 1 -- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e2c0052f0 msgr2=0x7f0e4c13fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e41d9b640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e2c0052f0 0x7f0e4c13fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e41d9b640 1 -- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e4c150ef0 msgr2=0x7f0e4c147df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e41d9b640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e4c150ef0 0x7f0e4c147df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e41d9b640 1 -- 172.21.15.62:0/233755204 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e4c0b56a0 con 0x7f0e4c1506a0 2024-04-18T10:06:06.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e4ad9d640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e4c150ef0 0x7f0e4c147df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:06.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e41d9b640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4c1506a0 0x7f0e4c1e3550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400d7d0 tx=0x7f0e3400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:06.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e437fe640 1 -- 172.21.15.62:0/233755204 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e34019070 con 0x7f0e4c1506a0 2024-04-18T10:06:06.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e437fe640 1 -- 172.21.15.62:0/233755204 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e340074e0 con 0x7f0e4c1506a0 2024-04-18T10:06:06.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 -- 172.21.15.62:0/233755204 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e4c14fbc0 con 0x7f0e4c1506a0 2024-04-18T10:06:06.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e437fe640 1 -- 172.21.15.62:0/233755204 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e340049f0 con 0x7f0e4c1506a0 2024-04-18T10:06:06.245 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.242+0000 7f0e50a5c640 1 -- 172.21.15.62:0/233755204 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e4c150180 con 0x7f0e4c1506a0 2024-04-18T10:06:06.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.246+0000 7f0e50a5c640 1 -- 172.21.15.62:0/233755204 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e4c150380 con 0x7f0e4c1506a0 2024-04-18T10:06:06.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.246+0000 7f0e437fe640 1 -- 172.21.15.62:0/233755204 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0e3401c070 con 0x7f0e4c1506a0 2024-04-18T10:06:06.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.246+0000 7f0e437fe640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0e30068380 0x7f0e3006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.246+0000 7f0e437fe640 1 -- 172.21.15.62:0/233755204 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0e34086280 con 0x7f0e4c1506a0 2024-04-18T10:06:06.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.246+0000 7f0e4a59c640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0e30068380 0x7f0e3006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.246 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.246+0000 7f0e4a59c640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0e30068380 0x7f0e3006a840 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c0098f0 tx=0x7f0e3c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:06.249 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.250+0000 7f0e437fe640 1 -- 172.21.15.62:0/233755204 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0e4c150380 con 0x7f0e4c1506a0 2024-04-18T10:06:06.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.406+0000 7f0e50a5c640 1 -- 172.21.15.62:0/233755204 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0e4c068b90 con 0x7f0e30068380 2024-04-18T10:06:06.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.422+0000 7f0e437fe640 1 -- 172.21.15.62:0/233755204 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f0e4c068b90 con 0x7f0e30068380 2024-04-18T10:06:06.424 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:06.425 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-18T10:06:06.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.426+0000 7f0e50a5c640 1 -- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0e30068380 msgr2=0x7f0e3006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.426+0000 7f0e50a5c640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0e30068380 0x7f0e3006a840 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0e3c0098f0 tx=0x7f0e3c008040 comp rx=0 tx=0).stop 2024-04-18T10:06:06.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.426+0000 7f0e50a5c640 1 -- 172.21.15.62:0/233755204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4c1506a0 msgr2=0x7f0e4c1e3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.426+0000 7f0e50a5c640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4c1506a0 0x7f0e4c1e3550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400d7d0 tx=0x7f0e3400dc90 comp rx=0 tx=0).stop 2024-04-18T10:06:06.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.426+0000 7f0e50a5c640 1 -- 172.21.15.62:0/233755204 shutdown_connections 2024-04-18T10:06:06.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.426+0000 7f0e50a5c640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0e30068380 0x7f0e3006a840 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.426+0000 7f0e50a5c640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e4c150ef0 0x7f0e4c147df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.426+0000 7f0e50a5c640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0e4c1506a0 0x7f0e4c1e3550 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.426+0000 7f0e50a5c640 1 --2- 172.21.15.62:0/233755204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e2c0052f0 0x7f0e4c13fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.426+0000 7f0e50a5c640 1 -- 172.21.15.62:0/233755204 >> 172.21.15.62:0/233755204 conn(0x7f0e4c06c0d0 msgr2=0x7f0e4c0b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:06.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.426+0000 7f0e50a5c640 1 -- 172.21.15.62:0/233755204 shutdown_connections 2024-04-18T10:06:06.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.426+0000 7f0e50a5c640 1 -- 172.21.15.62:0/233755204 wait complete. 2024-04-18T10:06:06.441 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-04-18T10:06:05.126999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219116,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755604,"statfs":{"total":773094113280,"available":772869738496,"internally_reserved":0,"allocated":2990080,"data_stored":1790396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.923340"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242380+0000","last_change":"2024-04-18T10:05:59.270038+0000","last_active":"2024-04-18T10:06:00.242380+0000","last_peered":"2024-04-18T10:06:00.242380+0000","last_clean":"2024-04-18T10:06:00.242380+0000","last_became_active":"2024-04-18T10:05:59.269226+0000","last_became_peered":"2024-04-18T10:05:59.269226+0000","last_unstale":"2024-04-18T10:06:00.242380+0000","last_undegraded":"2024-04-18T10:06:00.242380+0000","last_fullsized":"2024-04-18T10:06:00.242380+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:45:43.670227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232794+0000","last_change":"2024-04-18T10:05:59.230274+0000","last_active":"2024-04-18T10:06:00.232794+0000","last_peered":"2024-04-18T10:06:00.232794+0000","last_clean":"2024-04-18T10:06:00.232794+0000","last_became_active":"2024-04-18T10:05:59.229843+0000","last_became_peered":"2024-04-18T10:05:59.229843+0000","last_unstale":"2024-04-18T10:06:00.232794+0000","last_undegraded":"2024-04-18T10:06:00.232794+0000","last_fullsized":"2024-04-18T10:06:00.232794+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384628+0000","last_change":"2024-04-18T10:05:59.228858+0000","last_active":"2024-04-18T10:06:00.384628+0000","last_peered":"2024-04-18T10:06:00.384628+0000","last_clean":"2024-04-18T10:06:00.384628+0000","last_became_active":"2024-04-18T10:05:59.228382+0000","last_became_peered":"2024-04-18T10:05:59.228382+0000","last_unstale":"2024-04-18T10:06:00.384628+0000","last_undegraded":"2024-04-18T10:06:00.384628+0000","last_fullsized":"2024-04-18T10:06:00.384628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240458+0000","last_change":"2024-04-18T10:05:59.240142+0000","last_active":"2024-04-18T10:06:00.240458+0000","last_peered":"2024-04-18T10:06:00.240458+0000","last_clean":"2024-04-18T10:06:00.240458+0000","last_became_active":"2024-04-18T10:05:59.239298+0000","last_became_peered":"2024-04-18T10:05:59.239298+0000","last_unstale":"2024-04-18T10:06:00.240458+0000","last_undegraded":"2024-04-18T10:06:00.240458+0000","last_fullsized":"2024-04-18T10:06:00.240458+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:49.354229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239594+0000","last_change":"2024-04-18T10:05:59.235434+0000","last_active":"2024-04-18T10:06:00.239594+0000","last_peered":"2024-04-18T10:06:00.239594+0000","last_clean":"2024-04-18T10:06:00.239594+0000","last_became_active":"2024-04-18T10:05:59.234575+0000","last_became_peered":"2024-04-18T10:05:59.234575+0000","last_unstale":"2024-04-18T10:06:00.239594+0000","last_undegraded":"2024-04-18T10:06:00.239594+0000","last_fullsized":"2024-04-18T10:06:00.239594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:34:28.871228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238452+0000","last_change":"2024-04-18T10:05:59.232083+0000","last_active":"2024-04-18T10:06:00.238452+0000","last_peered":"2024-04-18T10:06:00.238452+0000","last_clean":"2024-04-18T10:06:00.238452+0000","last_became_active":"2024-04-18T10:05:59.231313+0000","last_became_peered":"2024-04-18T10:05:59.231313+0000","last_unstale":"2024-04-18T10:06:00.238452+0000","last_undegraded":"2024-04-18T10:06:00.238452+0000","last_fullsized":"2024-04-18T10:06:00.238452+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238876+0000","last_change":"2024-04-18T10:05:59.250962+0000","last_active":"2024-04-18T10:06:00.238876+0000","last_peered":"2024-04-18T10:06:00.238876+0000","last_clean":"2024-04-18T10:06:00.238876+0000","last_became_active":"2024-04-18T10:05:59.250059+0000","last_became_peered":"2024-04-18T10:05:59.250059+0000","last_unstale":"2024-04-18T10:06:00.238876+0000","last_undegraded":"2024-04-18T10:06:00.238876+0000","last_fullsized":"2024-04-18T10:06:00.238876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:12:17.167339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383437+0000","last_change":"2024-04-18T10:05:59.256210+0000","last_active":"2024-04-18T10:06:00.383437+0000","last_peered":"2024-04-18T10:06:00.383437+0000","last_clean":"2024-04-18T10:06:00.383437+0000","last_became_active":"2024-04-18T10:05:59.255975+0000","last_became_peered":"2024-04-18T10:05:59.255975+0000","last_unstale":"2024-04-18T10:06:00.383437+0000","last_undegraded":"2024-04-18T10:06:00.383437+0000","last_fullsized":"2024-04-18T10:06:00.383437+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:49:34.764977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242471+0000","last_change":"2024-04-18T10:05:59.272589+0000","last_active":"2024-04-18T10:06:00.242471+0000","last_peered":"2024-04-18T10:06:00.242471+0000","last_clean":"2024-04-18T10:06:00.242471+0000","last_became_active":"2024-04-18T10:05:59.271883+0000","last_became_peered":"2024-04-18T10:05:59.271883+0000","last_unstale":"2024-04-18T10:06:00.242471+0000","last_undegraded":"2024-04-18T10:06:00.242471+0000","last_fullsized":"2024-04-18T10:06:00.242471+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:24:15.929261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243230+0000","last_change":"2024-04-18T10:05:59.261146+0000","last_active":"2024-04-18T10:06:00.243230+0000","last_peered":"2024-04-18T10:06:00.243230+0000","last_clean":"2024-04-18T10:06:00.243230+0000","last_became_active":"2024-04-18T10:05:59.260455+0000","last_became_peered":"2024-04-18T10:05:59.260455+0000","last_unstale":"2024-04-18T10:06:00.243230+0000","last_undegraded":"2024-04-18T10:06:00.243230+0000","last_fullsized":"2024-04-18T10:06:00.243230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.244095+0000","last_change":"2024-04-18T10:05:59.260958+0000","last_active":"2024-04-18T10:06:00.244095+0000","last_peered":"2024-04-18T10:06:00.244095+0000","last_clean":"2024-04-18T10:06:00.244095+0000","last_became_active":"2024-04-18T10:05:59.260251+0000","last_became_peered":"2024-04-18T10:05:59.260251+0000","last_unstale":"2024-04-18T10:06:00.244095+0000","last_undegraded":"2024-04-18T10:06:00.244095+0000","last_fullsized":"2024-04-18T10:06:00.244095+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384238+0000","last_change":"2024-04-18T10:05:59.238407+0000","last_active":"2024-04-18T10:06:00.384238+0000","last_peered":"2024-04-18T10:06:00.384238+0000","last_clean":"2024-04-18T10:06:00.384238+0000","last_became_active":"2024-04-18T10:05:59.236530+0000","last_became_peered":"2024-04-18T10:05:59.236530+0000","last_unstale":"2024-04-18T10:06:00.384238+0000","last_undegraded":"2024-04-18T10:06:00.384238+0000","last_fullsized":"2024-04-18T10:06:00.384238+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:51.419833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384926+0000","last_change":"2024-04-18T10:05:59.243032+0000","last_active":"2024-04-18T10:06:00.384926+0000","last_peered":"2024-04-18T10:06:00.384926+0000","last_clean":"2024-04-18T10:06:00.384926+0000","last_became_active":"2024-04-18T10:05:59.242716+0000","last_became_peered":"2024-04-18T10:05:59.242716+0000","last_unstale":"2024-04-18T10:06:00.384926+0000","last_undegraded":"2024-04-18T10:06:00.384926+0000","last_fullsized":"2024-04-18T10:06:00.384926+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384052+0000","last_change":"2024-04-18T10:05:59.240754+0000","last_active":"2024-04-18T10:06:00.384052+0000","last_peered":"2024-04-18T10:06:00.384052+0000","last_clean":"2024-04-18T10:06:00.384052+0000","last_became_active":"2024-04-18T10:05:59.239992+0000","last_became_peered":"2024-04-18T10:05:59.239992+0000","last_unstale":"2024-04-18T10:06:00.384052+0000","last_undegraded":"2024-04-18T10:06:00.384052+0000","last_fullsized":"2024-04-18T10:06:00.384052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232972+0000","last_change":"2024-04-18T10:05:59.224563+0000","last_active":"2024-04-18T10:06:00.232972+0000","last_peered":"2024-04-18T10:06:00.232972+0000","last_clean":"2024-04-18T10:06:00.232972+0000","last_became_active":"2024-04-18T10:05:59.224115+0000","last_became_peered":"2024-04-18T10:05:59.224115+0000","last_unstale":"2024-04-18T10:06:00.232972+0000","last_undegraded":"2024-04-18T10:06:00.232972+0000","last_fullsized":"2024-04-18T10:06:00.232972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238461+0000","last_change":"2024-04-18T10:05:59.232777+0000","last_active":"2024-04-18T10:06:00.238461+0000","last_peered":"2024-04-18T10:06:00.238461+0000","last_clean":"2024-04-18T10:06:00.238461+0000","last_became_active":"2024-04-18T10:05:59.231188+0000","last_became_peered":"2024-04-18T10:05:59.231188+0000","last_unstale":"2024-04-18T10:06:00.238461+0000","last_undegraded":"2024-04-18T10:06:00.238461+0000","last_fullsized":"2024-04-18T10:06:00.238461+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243840+0000","last_change":"2024-04-18T10:05:59.264529+0000","last_active":"2024-04-18T10:06:00.243840+0000","last_peered":"2024-04-18T10:06:00.243840+0000","last_clean":"2024-04-18T10:06:00.243840+0000","last_became_active":"2024-04-18T10:05:59.260972+0000","last_became_peered":"2024-04-18T10:05:59.260972+0000","last_unstale":"2024-04-18T10:06:00.243840+0000","last_undegraded":"2024-04-18T10:06:00.243840+0000","last_fullsized":"2024-04-18T10:06:00.243840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:24:15.929261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385847+0000","last_change":"2024-04-18T10:05:59.262102+0000","last_active":"2024-04-18T10:06:00.385847+0000","last_peered":"2024-04-18T10:06:00.385847+0000","last_clean":"2024-04-18T10:06:00.385847+0000","last_became_active":"2024-04-18T10:05:59.261796+0000","last_became_peered":"2024-04-18T10:05:59.261796+0000","last_unstale":"2024-04-18T10:06:00.385847+0000","last_undegraded":"2024-04-18T10:06:00.385847+0000","last_fullsized":"2024-04-18T10:06:00.385847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:51.419833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241173+0000","last_change":"2024-04-18T10:05:59.256981+0000","last_active":"2024-04-18T10:06:00.241173+0000","last_peered":"2024-04-18T10:06:00.241173+0000","last_clean":"2024-04-18T10:06:00.241173+0000","last_became_active":"2024-04-18T10:05:59.252309+0000","last_became_peered":"2024-04-18T10:05:59.252309+0000","last_unstale":"2024-04-18T10:06:00.241173+0000","last_undegraded":"2024-04-18T10:06:00.241173+0000","last_fullsized":"2024-04-18T10:06:00.241173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:16:23.709955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243343+0000","last_change":"2024-04-18T10:05:59.266020+0000","last_active":"2024-04-18T10:06:00.243343+0000","last_peered":"2024-04-18T10:06:00.243343+0000","last_clean":"2024-04-18T10:06:00.243343+0000","last_became_active":"2024-04-18T10:05:59.265361+0000","last_became_peered":"2024-04-18T10:05:59.265361+0000","last_unstale":"2024-04-18T10:06:00.243343+0000","last_undegraded":"2024-04-18T10:06:00.243343+0000","last_fullsized":"2024-04-18T10:06:00.243343+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384730+0000","last_change":"2024-04-18T10:05:59.262190+0000","last_active":"2024-04-18T10:06:00.384730+0000","last_peered":"2024-04-18T10:06:00.384730+0000","last_clean":"2024-04-18T10:06:00.384730+0000","last_became_active":"2024-04-18T10:05:59.261866+0000","last_became_peered":"2024-04-18T10:05:59.261866+0000","last_unstale":"2024-04-18T10:06:00.384730+0000","last_undegraded":"2024-04-18T10:06:00.384730+0000","last_fullsized":"2024-04-18T10:06:00.384730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:49:34.764977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238819+0000","last_change":"2024-04-18T10:05:59.241285+0000","last_active":"2024-04-18T10:06:00.238819+0000","last_peered":"2024-04-18T10:06:00.238819+0000","last_clean":"2024-04-18T10:06:00.238819+0000","last_became_active":"2024-04-18T10:05:59.239568+0000","last_became_peered":"2024-04-18T10:05:59.239568+0000","last_unstale":"2024-04-18T10:06:00.238819+0000","last_undegraded":"2024-04-18T10:06:00.238819+0000","last_fullsized":"2024-04-18T10:06:00.238819+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:12:17.167339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385177+0000","last_change":"2024-04-18T10:05:59.264180+0000","last_active":"2024-04-18T10:06:00.385177+0000","last_peered":"2024-04-18T10:06:00.385177+0000","last_clean":"2024-04-18T10:06:00.385177+0000","last_became_active":"2024-04-18T10:05:59.263866+0000","last_became_peered":"2024-04-18T10:05:59.263866+0000","last_unstale":"2024-04-18T10:06:00.385177+0000","last_undegraded":"2024-04-18T10:06:00.385177+0000","last_fullsized":"2024-04-18T10:06:00.385177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238829+0000","last_change":"2024-04-18T10:05:59.246355+0000","last_active":"2024-04-18T10:06:00.238829+0000","last_peered":"2024-04-18T10:06:00.238829+0000","last_clean":"2024-04-18T10:06:00.238829+0000","last_became_active":"2024-04-18T10:05:59.239576+0000","last_became_peered":"2024-04-18T10:05:59.239576+0000","last_unstale":"2024-04-18T10:06:00.238829+0000","last_undegraded":"2024-04-18T10:06:00.238829+0000","last_fullsized":"2024-04-18T10:06:00.238829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:27:56.094004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386366+0000","last_change":"2024-04-18T10:05:59.264586+0000","last_active":"2024-04-18T10:06:00.386366+0000","last_peered":"2024-04-18T10:06:00.386366+0000","last_clean":"2024-04-18T10:06:00.386366+0000","last_became_active":"2024-04-18T10:05:59.264215+0000","last_became_peered":"2024-04-18T10:05:59.264215+0000","last_unstale":"2024-04-18T10:06:00.386366+0000","last_undegraded":"2024-04-18T10:06:00.386366+0000","last_fullsized":"2024-04-18T10:06:00.386366+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:27:56.094004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242998+0000","last_change":"2024-04-18T10:05:59.243781+0000","last_active":"2024-04-18T10:06:00.242998+0000","last_peered":"2024-04-18T10:06:00.242998+0000","last_clean":"2024-04-18T10:06:00.242998+0000","last_became_active":"2024-04-18T10:05:59.243092+0000","last_became_peered":"2024-04-18T10:05:59.243092+0000","last_unstale":"2024-04-18T10:06:00.242998+0000","last_undegraded":"2024-04-18T10:06:00.242998+0000","last_fullsized":"2024-04-18T10:06:00.242998+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:49:34.764977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232294+0000","last_change":"2024-04-18T10:05:59.234128+0000","last_active":"2024-04-18T10:06:00.232294+0000","last_peered":"2024-04-18T10:06:00.232294+0000","last_clean":"2024-04-18T10:06:00.232294+0000","last_became_active":"2024-04-18T10:05:59.233742+0000","last_became_peered":"2024-04-18T10:05:59.233742+0000","last_unstale":"2024-04-18T10:06:00.232294+0000","last_undegraded":"2024-04-18T10:06:00.232294+0000","last_fullsized":"2024-04-18T10:06:00.232294+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386224+0000","last_change":"2024-04-18T10:05:59.264826+0000","last_active":"2024-04-18T10:06:00.386224+0000","last_peered":"2024-04-18T10:06:00.386224+0000","last_clean":"2024-04-18T10:06:00.386224+0000","last_became_active":"2024-04-18T10:05:59.263655+0000","last_became_peered":"2024-04-18T10:05:59.263655+0000","last_unstale":"2024-04-18T10:06:00.386224+0000","last_undegraded":"2024-04-18T10:06:00.386224+0000","last_fullsized":"2024-04-18T10:06:00.386224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:12:17.167339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383983+0000","last_change":"2024-04-18T10:05:59.235032+0000","last_active":"2024-04-18T10:06:00.383983+0000","last_peered":"2024-04-18T10:06:00.383983+0000","last_clean":"2024-04-18T10:06:00.383983+0000","last_became_active":"2024-04-18T10:05:59.234660+0000","last_became_peered":"2024-04-18T10:05:59.234660+0000","last_unstale":"2024-04-18T10:06:00.383983+0000","last_undegraded":"2024-04-18T10:06:00.383983+0000","last_fullsized":"2024-04-18T10:06:00.383983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239182+0000","last_change":"2024-04-18T10:05:59.241915+0000","last_active":"2024-04-18T10:06:00.239182+0000","last_peered":"2024-04-18T10:06:00.239182+0000","last_clean":"2024-04-18T10:06:00.239182+0000","last_became_active":"2024-04-18T10:05:59.240519+0000","last_became_peered":"2024-04-18T10:05:59.240519+0000","last_unstale":"2024-04-18T10:06:00.239182+0000","last_undegraded":"2024-04-18T10:06:00.239182+0000","last_fullsized":"2024-04-18T10:06:00.239182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:34:28.871228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385003+0000","last_change":"2024-04-18T10:05:59.246146+0000","last_active":"2024-04-18T10:06:00.385003+0000","last_peered":"2024-04-18T10:06:00.385003+0000","last_clean":"2024-04-18T10:06:00.385003+0000","last_became_active":"2024-04-18T10:05:59.245800+0000","last_became_peered":"2024-04-18T10:05:59.245800+0000","last_unstale":"2024-04-18T10:06:00.385003+0000","last_undegraded":"2024-04-18T10:06:00.385003+0000","last_fullsized":"2024-04-18T10:06:00.385003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:22.118304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242883+0000","last_change":"2024-04-18T10:05:59.239684+0000","last_active":"2024-04-18T10:06:00.242883+0000","last_peered":"2024-04-18T10:06:00.242883+0000","last_clean":"2024-04-18T10:06:00.242883+0000","last_became_active":"2024-04-18T10:05:59.233494+0000","last_became_peered":"2024-04-18T10:05:59.233494+0000","last_unstale":"2024-04-18T10:06:00.242883+0000","last_undegraded":"2024-04-18T10:06:00.242883+0000","last_fullsized":"2024-04-18T10:06:00.242883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:23:54.569394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383451+0000","last_change":"2024-04-18T10:05:59.219492+0000","last_active":"2024-04-18T10:06:00.383451+0000","last_peered":"2024-04-18T10:06:00.383451+0000","last_clean":"2024-04-18T10:06:00.383451+0000","last_became_active":"2024-04-18T10:05:59.218737+0000","last_became_peered":"2024-04-18T10:05:59.218737+0000","last_unstale":"2024-04-18T10:06:00.383451+0000","last_undegraded":"2024-04-18T10:06:00.383451+0000","last_fullsized":"2024-04-18T10:06:00.383451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:33.010147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386355+0000","last_change":"2024-04-18T10:05:58.253056+0000","last_active":"2024-04-18T10:06:00.386355+0000","last_peered":"2024-04-18T10:06:00.386355+0000","last_clean":"2024-04-18T10:06:00.386355+0000","last_became_active":"2024-04-18T10:05:58.252737+0000","last_became_peered":"2024-04-18T10:05:58.252737+0000","last_unstale":"2024-04-18T10:06:00.386355+0000","last_undegraded":"2024-04-18T10:06:00.386355+0000","last_fullsized":"2024-04-18T10:06:00.386355+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384430+0000","last_change":"2024-04-18T10:05:59.263750+0000","last_active":"2024-04-18T10:06:00.384430+0000","last_peered":"2024-04-18T10:06:00.384430+0000","last_clean":"2024-04-18T10:06:00.384430+0000","last_became_active":"2024-04-18T10:05:59.263538+0000","last_became_peered":"2024-04-18T10:05:59.263538+0000","last_unstale":"2024-04-18T10:06:00.384430+0000","last_undegraded":"2024-04-18T10:06:00.384430+0000","last_fullsized":"2024-04-18T10:06:00.384430+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243880+0000","last_change":"2024-04-18T10:05:58.243026+0000","last_active":"2024-04-18T10:06:00.243880+0000","last_peered":"2024-04-18T10:06:00.243880+0000","last_clean":"2024-04-18T10:06:00.243880+0000","last_became_active":"2024-04-18T10:05:58.240506+0000","last_became_peered":"2024-04-18T10:05:58.240506+0000","last_unstale":"2024-04-18T10:06:00.243880+0000","last_undegraded":"2024-04-18T10:06:00.243880+0000","last_fullsized":"2024-04-18T10:06:00.243880+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385279+0000","last_change":"2024-04-18T10:05:58.243128+0000","last_active":"2024-04-18T10:06:00.385279+0000","last_peered":"2024-04-18T10:06:00.385279+0000","last_clean":"2024-04-18T10:06:00.385279+0000","last_became_active":"2024-04-18T10:05:58.242694+0000","last_became_peered":"2024-04-18T10:05:58.242694+0000","last_unstale":"2024-04-18T10:06:00.385279+0000","last_undegraded":"2024-04-18T10:06:00.385279+0000","last_fullsized":"2024-04-18T10:06:00.385279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:52:26.587547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383890+0000","last_change":"2024-04-18T10:05:59.232704+0000","last_active":"2024-04-18T10:06:00.383890+0000","last_peered":"2024-04-18T10:06:00.383890+0000","last_clean":"2024-04-18T10:06:00.383890+0000","last_became_active":"2024-04-18T10:05:59.231762+0000","last_became_peered":"2024-04-18T10:05:59.231762+0000","last_unstale":"2024-04-18T10:06:00.383890+0000","last_undegraded":"2024-04-18T10:06:00.383890+0000","last_fullsized":"2024-04-18T10:06:00.383890+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239055+0000","last_change":"2024-04-18T10:05:58.231745+0000","last_active":"2024-04-18T10:06:00.239055+0000","last_peered":"2024-04-18T10:06:00.239055+0000","last_clean":"2024-04-18T10:06:00.239055+0000","last_became_active":"2024-04-18T10:05:58.230994+0000","last_became_peered":"2024-04-18T10:05:58.230994+0000","last_unstale":"2024-04-18T10:06:00.239055+0000","last_undegraded":"2024-04-18T10:06:00.239055+0000","last_fullsized":"2024-04-18T10:06:00.239055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232217+0000","last_change":"2024-04-18T10:05:59.232550+0000","last_active":"2024-04-18T10:06:00.232217+0000","last_peered":"2024-04-18T10:06:00.232217+0000","last_clean":"2024-04-18T10:06:00.232217+0000","last_became_active":"2024-04-18T10:05:59.231955+0000","last_became_peered":"2024-04-18T10:05:59.231955+0000","last_unstale":"2024-04-18T10:06:00.232217+0000","last_undegraded":"2024-04-18T10:06:00.232217+0000","last_fullsized":"2024-04-18T10:06:00.232217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243516+0000","last_change":"2024-04-18T10:05:58.244903+0000","last_active":"2024-04-18T10:06:00.243516+0000","last_peered":"2024-04-18T10:06:00.243516+0000","last_clean":"2024-04-18T10:06:00.243516+0000","last_became_active":"2024-04-18T10:05:58.244287+0000","last_became_peered":"2024-04-18T10:05:58.244287+0000","last_unstale":"2024-04-18T10:06:00.243516+0000","last_undegraded":"2024-04-18T10:06:00.243516+0000","last_fullsized":"2024-04-18T10:06:00.243516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240811+0000","last_change":"2024-04-18T10:05:59.238308+0000","last_active":"2024-04-18T10:06:00.240811+0000","last_peered":"2024-04-18T10:06:00.240811+0000","last_clean":"2024-04-18T10:06:00.240811+0000","last_became_active":"2024-04-18T10:05:59.237422+0000","last_became_peered":"2024-04-18T10:05:59.237422+0000","last_unstale":"2024-04-18T10:06:00.240811+0000","last_undegraded":"2024-04-18T10:06:00.240811+0000","last_fullsized":"2024-04-18T10:06:00.240811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:28:59.797065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240802+0000","last_change":"2024-04-18T10:05:58.245925+0000","last_active":"2024-04-18T10:06:00.240802+0000","last_peered":"2024-04-18T10:06:00.240802+0000","last_clean":"2024-04-18T10:06:00.240802+0000","last_became_active":"2024-04-18T10:05:58.244713+0000","last_became_peered":"2024-04-18T10:05:58.244713+0000","last_unstale":"2024-04-18T10:06:00.240802+0000","last_undegraded":"2024-04-18T10:06:00.240802+0000","last_fullsized":"2024-04-18T10:06:00.240802+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384230+0000","last_change":"2024-04-18T10:05:59.252961+0000","last_active":"2024-04-18T10:06:00.384230+0000","last_peered":"2024-04-18T10:06:00.384230+0000","last_clean":"2024-04-18T10:06:00.384230+0000","last_became_active":"2024-04-18T10:05:59.252672+0000","last_became_peered":"2024-04-18T10:05:59.252672+0000","last_unstale":"2024-04-18T10:06:00.384230+0000","last_undegraded":"2024-04-18T10:06:00.384230+0000","last_fullsized":"2024-04-18T10:06:00.384230+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232406+0000","last_change":"2024-04-18T10:05:58.235997+0000","last_active":"2024-04-18T10:06:00.232406+0000","last_peered":"2024-04-18T10:06:00.232406+0000","last_clean":"2024-04-18T10:06:00.232406+0000","last_became_active":"2024-04-18T10:05:58.235330+0000","last_became_peered":"2024-04-18T10:05:58.235330+0000","last_unstale":"2024-04-18T10:06:00.232406+0000","last_undegraded":"2024-04-18T10:06:00.232406+0000","last_fullsized":"2024-04-18T10:06:00.232406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:21.107270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384433+0000","last_change":"2024-04-18T10:05:59.257176+0000","last_active":"2024-04-18T10:06:00.384433+0000","last_peered":"2024-04-18T10:06:00.384433+0000","last_clean":"2024-04-18T10:06:00.384433+0000","last_became_active":"2024-04-18T10:05:59.256452+0000","last_became_peered":"2024-04-18T10:05:59.256452+0000","last_unstale":"2024-04-18T10:06:00.384433+0000","last_undegraded":"2024-04-18T10:06:00.384433+0000","last_fullsized":"2024-04-18T10:06:00.384433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:04:27.662085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243463+0000","last_change":"2024-04-18T10:05:58.239207+0000","last_active":"2024-04-18T10:06:00.243463+0000","last_peered":"2024-04-18T10:06:00.243463+0000","last_clean":"2024-04-18T10:06:00.243463+0000","last_became_active":"2024-04-18T10:05:58.238349+0000","last_became_peered":"2024-04-18T10:05:58.238349+0000","last_unstale":"2024-04-18T10:06:00.243463+0000","last_undegraded":"2024-04-18T10:06:00.243463+0000","last_fullsized":"2024-04-18T10:06:00.243463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239372+0000","last_change":"2024-04-18T10:05:59.254087+0000","last_active":"2024-04-18T10:06:00.239372+0000","last_peered":"2024-04-18T10:06:00.239372+0000","last_clean":"2024-04-18T10:06:00.239372+0000","last_became_active":"2024-04-18T10:05:59.253296+0000","last_became_peered":"2024-04-18T10:05:59.253296+0000","last_unstale":"2024-04-18T10:06:00.239372+0000","last_undegraded":"2024-04-18T10:06:00.239372+0000","last_fullsized":"2024-04-18T10:06:00.239372+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:27:56.094004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384701+0000","last_change":"2024-04-18T10:05:58.236375+0000","last_active":"2024-04-18T10:06:00.384701+0000","last_peered":"2024-04-18T10:06:00.384701+0000","last_clean":"2024-04-18T10:06:00.384701+0000","last_became_active":"2024-04-18T10:05:58.236066+0000","last_became_peered":"2024-04-18T10:05:58.236066+0000","last_unstale":"2024-04-18T10:06:00.384701+0000","last_undegraded":"2024-04-18T10:06:00.384701+0000","last_fullsized":"2024-04-18T10:06:00.384701+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:52:26.587547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383621+0000","last_change":"2024-04-18T10:05:59.262667+0000","last_active":"2024-04-18T10:06:00.383621+0000","last_peered":"2024-04-18T10:06:00.383621+0000","last_clean":"2024-04-18T10:06:00.383621+0000","last_became_active":"2024-04-18T10:05:59.262365+0000","last_became_peered":"2024-04-18T10:05:59.262365+0000","last_unstale":"2024-04-18T10:06:00.383621+0000","last_undegraded":"2024-04-18T10:06:00.383621+0000","last_fullsized":"2024-04-18T10:06:00.383621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:23:54.569394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239431+0000","last_change":"2024-04-18T10:05:58.243114+0000","last_active":"2024-04-18T10:06:00.239431+0000","last_peered":"2024-04-18T10:06:00.239431+0000","last_clean":"2024-04-18T10:06:00.239431+0000","last_became_active":"2024-04-18T10:05:58.242102+0000","last_became_peered":"2024-04-18T10:05:58.242102+0000","last_unstale":"2024-04-18T10:06:00.239431+0000","last_undegraded":"2024-04-18T10:06:00.239431+0000","last_fullsized":"2024-04-18T10:06:00.239431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239878+0000","last_change":"2024-04-18T10:05:59.274245+0000","last_active":"2024-04-18T10:06:00.239878+0000","last_peered":"2024-04-18T10:06:00.239878+0000","last_clean":"2024-04-18T10:06:00.239878+0000","last_became_active":"2024-04-18T10:05:59.273707+0000","last_became_peered":"2024-04-18T10:05:59.273707+0000","last_unstale":"2024-04-18T10:06:00.239878+0000","last_undegraded":"2024-04-18T10:06:00.239878+0000","last_fullsized":"2024-04-18T10:06:00.239878+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:49.354229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243436+0000","last_change":"2024-04-18T10:05:58.242951+0000","last_active":"2024-04-18T10:06:00.243436+0000","last_peered":"2024-04-18T10:06:00.243436+0000","last_clean":"2024-04-18T10:06:00.243436+0000","last_became_active":"2024-04-18T10:05:58.241438+0000","last_became_peered":"2024-04-18T10:05:58.241438+0000","last_unstale":"2024-04-18T10:06:00.243436+0000","last_undegraded":"2024-04-18T10:06:00.243436+0000","last_fullsized":"2024-04-18T10:06:00.243436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243154+0000","last_change":"2024-04-18T10:05:59.233588+0000","last_active":"2024-04-18T10:06:00.243154+0000","last_peered":"2024-04-18T10:06:00.243154+0000","last_clean":"2024-04-18T10:06:00.243154+0000","last_became_active":"2024-04-18T10:05:59.232831+0000","last_became_peered":"2024-04-18T10:05:59.232831+0000","last_unstale":"2024-04-18T10:06:00.243154+0000","last_undegraded":"2024-04-18T10:06:00.243154+0000","last_fullsized":"2024-04-18T10:06:00.243154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:54:18.056379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384620+0000","last_change":"2024-04-18T10:05:58.223796+0000","last_active":"2024-04-18T10:06:00.384620+0000","last_peered":"2024-04-18T10:06:00.384620+0000","last_clean":"2024-04-18T10:06:00.384620+0000","last_became_active":"2024-04-18T10:05:58.223248+0000","last_became_peered":"2024-04-18T10:05:58.223248+0000","last_unstale":"2024-04-18T10:06:00.384620+0000","last_undegraded":"2024-04-18T10:06:00.384620+0000","last_fullsized":"2024-04-18T10:06:00.384620+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:52:26.587547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240028+0000","last_change":"2024-04-18T10:05:59.277176+0000","last_active":"2024-04-18T10:06:00.240028+0000","last_peered":"2024-04-18T10:06:00.240028+0000","last_clean":"2024-04-18T10:06:00.240028+0000","last_became_active":"2024-04-18T10:05:59.276386+0000","last_became_peered":"2024-04-18T10:05:59.276386+0000","last_unstale":"2024-04-18T10:06:00.240028+0000","last_undegraded":"2024-04-18T10:06:00.240028+0000","last_fullsized":"2024-04-18T10:06:00.240028+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:28:59.797065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384638+0000","last_change":"2024-04-18T10:05:58.233338+0000","last_active":"2024-04-18T10:06:00.384638+0000","last_peered":"2024-04-18T10:06:00.384638+0000","last_clean":"2024-04-18T10:06:00.384638+0000","last_became_active":"2024-04-18T10:05:58.231906+0000","last_became_peered":"2024-04-18T10:05:58.231906+0000","last_unstale":"2024-04-18T10:06:00.384638+0000","last_undegraded":"2024-04-18T10:06:00.384638+0000","last_fullsized":"2024-04-18T10:06:00.384638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:06:57.988393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241935+0000","last_change":"2024-04-18T10:05:59.275977+0000","last_active":"2024-04-18T10:06:00.241935+0000","last_peered":"2024-04-18T10:06:00.241935+0000","last_clean":"2024-04-18T10:06:00.241935+0000","last_became_active":"2024-04-18T10:05:59.275386+0000","last_became_peered":"2024-04-18T10:05:59.275386+0000","last_unstale":"2024-04-18T10:06:00.241935+0000","last_undegraded":"2024-04-18T10:06:00.241935+0000","last_fullsized":"2024-04-18T10:06:00.241935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:58:16.367066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386132+0000","last_change":"2024-04-18T10:05:58.229897+0000","last_active":"2024-04-18T10:06:00.386132+0000","last_peered":"2024-04-18T10:06:00.386132+0000","last_clean":"2024-04-18T10:06:00.386132+0000","last_became_active":"2024-04-18T10:05:58.229451+0000","last_became_peered":"2024-04-18T10:05:58.229451+0000","last_unstale":"2024-04-18T10:06:00.386132+0000","last_undegraded":"2024-04-18T10:06:00.386132+0000","last_fullsized":"2024-04-18T10:06:00.386132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242763+0000","last_change":"2024-04-18T10:05:59.276815+0000","last_active":"2024-04-18T10:06:00.242763+0000","last_peered":"2024-04-18T10:06:00.242763+0000","last_clean":"2024-04-18T10:06:00.242763+0000","last_became_active":"2024-04-18T10:05:59.276272+0000","last_became_peered":"2024-04-18T10:05:59.276272+0000","last_unstale":"2024-04-18T10:06:00.242763+0000","last_undegraded":"2024-04-18T10:06:00.242763+0000","last_fullsized":"2024-04-18T10:06:00.242763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:44:45.775310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232479+0000","last_change":"2024-04-18T10:05:58.230942+0000","last_active":"2024-04-18T10:06:00.232479+0000","last_peered":"2024-04-18T10:06:00.232479+0000","last_clean":"2024-04-18T10:06:00.232479+0000","last_became_active":"2024-04-18T10:05:58.230373+0000","last_became_peered":"2024-04-18T10:05:58.230373+0000","last_unstale":"2024-04-18T10:06:00.232479+0000","last_undegraded":"2024-04-18T10:06:00.232479+0000","last_fullsized":"2024-04-18T10:06:00.232479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:50.408798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241359+0000","last_change":"2024-04-18T10:05:59.234199+0000","last_active":"2024-04-18T10:06:00.241359+0000","last_peered":"2024-04-18T10:06:00.241359+0000","last_clean":"2024-04-18T10:06:00.241359+0000","last_became_active":"2024-04-18T10:05:59.233412+0000","last_became_peered":"2024-04-18T10:05:59.233412+0000","last_unstale":"2024-04-18T10:06:00.241359+0000","last_undegraded":"2024-04-18T10:06:00.241359+0000","last_fullsized":"2024-04-18T10:06:00.241359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:16:23.709955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385242+0000","last_change":"2024-04-18T10:05:58.222111+0000","last_active":"2024-04-18T10:06:00.385242+0000","last_peered":"2024-04-18T10:06:00.385242+0000","last_clean":"2024-04-18T10:06:00.385242+0000","last_became_active":"2024-04-18T10:05:58.221783+0000","last_became_peered":"2024-04-18T10:05:58.221783+0000","last_unstale":"2024-04-18T10:06:00.385242+0000","last_undegraded":"2024-04-18T10:06:00.385242+0000","last_fullsized":"2024-04-18T10:06:00.385242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240501+0000","last_change":"2024-04-18T10:05:59.261884+0000","last_active":"2024-04-18T10:06:00.240501+0000","last_peered":"2024-04-18T10:06:00.240501+0000","last_clean":"2024-04-18T10:06:00.240501+0000","last_became_active":"2024-04-18T10:05:59.261147+0000","last_became_peered":"2024-04-18T10:05:59.261147+0000","last_unstale":"2024-04-18T10:06:00.240501+0000","last_undegraded":"2024-04-18T10:06:00.240501+0000","last_fullsized":"2024-04-18T10:06:00.240501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:44:42.861177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238296+0000","last_change":"2024-04-18T10:05:49.183850+0000","last_active":"2024-04-18T10:06:00.238296+0000","last_peered":"2024-04-18T10:06:00.238296+0000","last_clean":"2024-04-18T10:06:00.238296+0000","last_became_active":"2024-04-18T10:05:49.181471+0000","last_became_peered":"2024-04-18T10:05:49.181471+0000","last_unstale":"2024-04-18T10:06:00.238296+0000","last_undegraded":"2024-04-18T10:06:00.238296+0000","last_fullsized":"2024-04-18T10:06:00.238296+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_clean_scrub_stamp":"2024-04-18T10:05:47.511486+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:06:51.214547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239978+0000","last_change":"2024-04-18T10:05:58.228257+0000","last_active":"2024-04-18T10:06:00.239978+0000","last_peered":"2024-04-18T10:06:00.239978+0000","last_clean":"2024-04-18T10:06:00.239978+0000","last_became_active":"2024-04-18T10:05:58.224594+0000","last_became_peered":"2024-04-18T10:05:58.224594+0000","last_unstale":"2024-04-18T10:06:00.239978+0000","last_undegraded":"2024-04-18T10:06:00.239978+0000","last_fullsized":"2024-04-18T10:06:00.239978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:42:47.391436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385522+0000","last_change":"2024-04-18T10:05:59.256229+0000","last_active":"2024-04-18T10:06:00.385522+0000","last_peered":"2024-04-18T10:06:00.385522+0000","last_clean":"2024-04-18T10:06:00.385522+0000","last_became_active":"2024-04-18T10:05:59.255894+0000","last_became_peered":"2024-04-18T10:05:59.255894+0000","last_unstale":"2024-04-18T10:06:00.385522+0000","last_undegraded":"2024-04-18T10:06:00.385522+0000","last_fullsized":"2024-04-18T10:06:00.385522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:25:25.334660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238224+0000","last_change":"2024-04-18T10:05:58.254200+0000","last_active":"2024-04-18T10:06:00.238224+0000","last_peered":"2024-04-18T10:06:00.238224+0000","last_clean":"2024-04-18T10:06:00.238224+0000","last_became_active":"2024-04-18T10:05:58.253365+0000","last_became_peered":"2024-04-18T10:05:58.253365+0000","last_unstale":"2024-04-18T10:06:00.238224+0000","last_undegraded":"2024-04-18T10:06:00.238224+0000","last_fullsized":"2024-04-18T10:06:00.238224+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383217+0000","last_change":"2024-04-18T10:05:59.264229+0000","last_active":"2024-04-18T10:06:00.383217+0000","last_peered":"2024-04-18T10:06:00.383217+0000","last_clean":"2024-04-18T10:06:00.383217+0000","last_became_active":"2024-04-18T10:05:59.263577+0000","last_became_peered":"2024-04-18T10:05:59.263577+0000","last_unstale":"2024-04-18T10:06:00.383217+0000","last_undegraded":"2024-04-18T10:06:00.383217+0000","last_fullsized":"2024-04-18T10:06:00.383217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:37:20.126666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385257+0000","last_change":"2024-04-18T10:05:58.235466+0000","last_active":"2024-04-18T10:06:00.385257+0000","last_peered":"2024-04-18T10:06:00.385257+0000","last_clean":"2024-04-18T10:06:00.385257+0000","last_became_active":"2024-04-18T10:05:58.234823+0000","last_became_peered":"2024-04-18T10:05:58.234823+0000","last_unstale":"2024-04-18T10:06:00.385257+0000","last_undegraded":"2024-04-18T10:06:00.385257+0000","last_fullsized":"2024-04-18T10:06:00.385257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232451+0000","last_change":"2024-04-18T10:05:59.230409+0000","last_active":"2024-04-18T10:06:00.232451+0000","last_peered":"2024-04-18T10:06:00.232451+0000","last_clean":"2024-04-18T10:06:00.232451+0000","last_became_active":"2024-04-18T10:05:59.228899+0000","last_became_peered":"2024-04-18T10:05:59.228899+0000","last_unstale":"2024-04-18T10:06:00.232451+0000","last_undegraded":"2024-04-18T10:06:00.232451+0000","last_fullsized":"2024-04-18T10:06:00.232451+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:21:07.686121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239789+0000","last_change":"2024-04-18T10:05:58.232028+0000","last_active":"2024-04-18T10:06:00.239789+0000","last_peered":"2024-04-18T10:06:00.239789+0000","last_clean":"2024-04-18T10:06:00.239789+0000","last_became_active":"2024-04-18T10:05:58.229311+0000","last_became_peered":"2024-04-18T10:05:58.229311+0000","last_unstale":"2024-04-18T10:06:00.239789+0000","last_undegraded":"2024-04-18T10:06:00.239789+0000","last_fullsized":"2024-04-18T10:06:00.239789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383872+0000","last_change":"2024-04-18T10:05:59.248629+0000","last_active":"2024-04-18T10:06:00.383872+0000","last_peered":"2024-04-18T10:06:00.383872+0000","last_clean":"2024-04-18T10:06:00.383872+0000","last_became_active":"2024-04-18T10:05:59.247619+0000","last_became_peered":"2024-04-18T10:05:59.247619+0000","last_unstale":"2024-04-18T10:06:00.383872+0000","last_undegraded":"2024-04-18T10:06:00.383872+0000","last_fullsized":"2024-04-18T10:06:00.383872+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:35.965832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232588+0000","last_change":"2024-04-18T10:05:58.224570+0000","last_active":"2024-04-18T10:06:00.232588+0000","last_peered":"2024-04-18T10:06:00.232588+0000","last_clean":"2024-04-18T10:06:00.232588+0000","last_became_active":"2024-04-18T10:05:58.224216+0000","last_became_peered":"2024-04-18T10:05:58.224216+0000","last_unstale":"2024-04-18T10:06:00.232588+0000","last_undegraded":"2024-04-18T10:06:00.232588+0000","last_fullsized":"2024-04-18T10:06:00.232588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:16:34.954797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242688+0000","last_change":"2024-04-18T10:05:59.246784+0000","last_active":"2024-04-18T10:06:00.242688+0000","last_peered":"2024-04-18T10:06:00.242688+0000","last_clean":"2024-04-18T10:06:00.242688+0000","last_became_active":"2024-04-18T10:05:59.243518+0000","last_became_peered":"2024-04-18T10:05:59.243518+0000","last_unstale":"2024-04-18T10:06:00.242688+0000","last_undegraded":"2024-04-18T10:06:00.242688+0000","last_fullsized":"2024-04-18T10:06:00.242688+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:45:43.670227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241271+0000","last_change":"2024-04-18T10:05:58.258428+0000","last_active":"2024-04-18T10:06:00.241271+0000","last_peered":"2024-04-18T10:06:00.241271+0000","last_clean":"2024-04-18T10:06:00.241271+0000","last_became_active":"2024-04-18T10:05:58.257648+0000","last_became_peered":"2024-04-18T10:05:58.257648+0000","last_unstale":"2024-04-18T10:06:00.241271+0000","last_undegraded":"2024-04-18T10:06:00.241271+0000","last_fullsized":"2024-04-18T10:06:00.241271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:15:05.853314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232543+0000","last_change":"2024-04-18T10:05:59.229368+0000","last_active":"2024-04-18T10:06:00.232543+0000","last_peered":"2024-04-18T10:06:00.232543+0000","last_clean":"2024-04-18T10:06:00.232543+0000","last_became_active":"2024-04-18T10:05:59.228912+0000","last_became_peered":"2024-04-18T10:05:59.228912+0000","last_unstale":"2024-04-18T10:06:00.232543+0000","last_undegraded":"2024-04-18T10:06:00.232543+0000","last_fullsized":"2024-04-18T10:06:00.232543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241350+0000","last_change":"2024-04-18T10:05:58.243743+0000","last_active":"2024-04-18T10:06:00.241350+0000","last_peered":"2024-04-18T10:06:00.241350+0000","last_clean":"2024-04-18T10:06:00.241350+0000","last_became_active":"2024-04-18T10:05:58.243448+0000","last_became_peered":"2024-04-18T10:05:58.243448+0000","last_unstale":"2024-04-18T10:06:00.241350+0000","last_undegraded":"2024-04-18T10:06:00.241350+0000","last_fullsized":"2024-04-18T10:06:00.241350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:04:26.651051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386763+0000","last_change":"2024-04-18T10:05:59.262383+0000","last_active":"2024-04-18T10:06:00.386763+0000","last_peered":"2024-04-18T10:06:00.386763+0000","last_clean":"2024-04-18T10:06:00.386763+0000","last_became_active":"2024-04-18T10:05:59.262039+0000","last_became_peered":"2024-04-18T10:05:59.262039+0000","last_unstale":"2024-04-18T10:06:00.386763+0000","last_undegraded":"2024-04-18T10:06:00.386763+0000","last_fullsized":"2024-04-18T10:06:00.386763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:49.354229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243760+0000","last_change":"2024-04-18T10:05:58.251779+0000","last_active":"2024-04-18T10:06:00.243760+0000","last_peered":"2024-04-18T10:06:00.243760+0000","last_clean":"2024-04-18T10:06:00.243760+0000","last_became_active":"2024-04-18T10:05:58.251062+0000","last_became_peered":"2024-04-18T10:05:58.251062+0000","last_unstale":"2024-04-18T10:06:00.243760+0000","last_undegraded":"2024-04-18T10:06:00.243760+0000","last_fullsized":"2024-04-18T10:06:00.243760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384029+0000","last_change":"2024-04-18T10:05:59.240684+0000","last_active":"2024-04-18T10:06:00.384029+0000","last_peered":"2024-04-18T10:06:00.384029+0000","last_clean":"2024-04-18T10:06:00.384029+0000","last_became_active":"2024-04-18T10:05:59.240231+0000","last_became_peered":"2024-04-18T10:05:59.240231+0000","last_unstale":"2024-04-18T10:06:00.384029+0000","last_undegraded":"2024-04-18T10:06:00.384029+0000","last_fullsized":"2024-04-18T10:06:00.384029+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.238119+0000","last_change":"2024-04-18T10:05:58.254872+0000","last_active":"2024-04-18T10:06:00.238119+0000","last_peered":"2024-04-18T10:06:00.238119+0000","last_clean":"2024-04-18T10:06:00.238119+0000","last_became_active":"2024-04-18T10:05:58.252439+0000","last_became_peered":"2024-04-18T10:05:58.252439+0000","last_unstale":"2024-04-18T10:06:00.238119+0000","last_undegraded":"2024-04-18T10:06:00.238119+0000","last_fullsized":"2024-04-18T10:06:00.238119+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384090+0000","last_change":"2024-04-18T10:05:59.254583+0000","last_active":"2024-04-18T10:06:00.384090+0000","last_peered":"2024-04-18T10:06:00.384090+0000","last_clean":"2024-04-18T10:06:00.384090+0000","last_became_active":"2024-04-18T10:05:59.253836+0000","last_became_peered":"2024-04-18T10:05:59.253836+0000","last_unstale":"2024-04-18T10:06:00.384090+0000","last_undegraded":"2024-04-18T10:06:00.384090+0000","last_fullsized":"2024-04-18T10:06:00.384090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:22.118304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232357+0000","last_change":"2024-04-18T10:05:58.248261+0000","last_active":"2024-04-18T10:06:00.232357+0000","last_peered":"2024-04-18T10:06:00.232357+0000","last_clean":"2024-04-18T10:06:00.232357+0000","last_became_active":"2024-04-18T10:05:58.247708+0000","last_became_peered":"2024-04-18T10:05:58.247708+0000","last_unstale":"2024-04-18T10:06:00.232357+0000","last_undegraded":"2024-04-18T10:06:00.232357+0000","last_fullsized":"2024-04-18T10:06:00.232357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:30:31.999112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.384524+0000","last_change":"2024-04-18T10:05:59.238345+0000","last_active":"2024-04-18T10:06:00.384524+0000","last_peered":"2024-04-18T10:06:00.384524+0000","last_clean":"2024-04-18T10:06:00.384524+0000","last_became_active":"2024-04-18T10:05:59.237905+0000","last_became_peered":"2024-04-18T10:05:59.237905+0000","last_unstale":"2024-04-18T10:06:00.384524+0000","last_undegraded":"2024-04-18T10:06:00.384524+0000","last_fullsized":"2024-04-18T10:06:00.384524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T11:26:48.259225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.232345+0000","last_change":"2024-04-18T10:05:58.229628+0000","last_active":"2024-04-18T10:06:00.232345+0000","last_peered":"2024-04-18T10:06:00.232345+0000","last_clean":"2024-04-18T10:06:00.232345+0000","last_became_active":"2024-04-18T10:05:58.229323+0000","last_became_peered":"2024-04-18T10:05:58.229323+0000","last_unstale":"2024-04-18T10:06:00.232345+0000","last_undegraded":"2024-04-18T10:06:00.232345+0000","last_fullsized":"2024-04-18T10:06:00.232345+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:55:20.627668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.385760+0000","last_change":"2024-04-18T10:05:59.232981+0000","last_active":"2024-04-18T10:06:00.385760+0000","last_peered":"2024-04-18T10:06:00.385760+0000","last_clean":"2024-04-18T10:06:00.385760+0000","last_became_active":"2024-04-18T10:05:59.232662+0000","last_became_peered":"2024-04-18T10:05:59.232662+0000","last_unstale":"2024-04-18T10:06:00.385760+0000","last_undegraded":"2024-04-18T10:06:00.385760+0000","last_fullsized":"2024-04-18T10:06:00.385760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:57:19.320239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239926+0000","last_change":"2024-04-18T10:05:58.258738+0000","last_active":"2024-04-18T10:06:00.239926+0000","last_peered":"2024-04-18T10:06:00.239926+0000","last_clean":"2024-04-18T10:06:00.239926+0000","last_became_active":"2024-04-18T10:05:58.257527+0000","last_became_peered":"2024-04-18T10:05:58.257527+0000","last_unstale":"2024-04-18T10:06:00.239926+0000","last_undegraded":"2024-04-18T10:06:00.239926+0000","last_fullsized":"2024-04-18T10:06:00.239926+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:39:21.107270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.243622+0000","last_change":"2024-04-18T10:05:59.264420+0000","last_active":"2024-04-18T10:06:00.243622+0000","last_peered":"2024-04-18T10:06:00.243622+0000","last_clean":"2024-04-18T10:06:00.243622+0000","last_became_active":"2024-04-18T10:05:59.263738+0000","last_became_peered":"2024-04-18T10:05:59.263738+0000","last_unstale":"2024-04-18T10:06:00.243622+0000","last_undegraded":"2024-04-18T10:06:00.243622+0000","last_fullsized":"2024-04-18T10:06:00.243622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:17:15.337857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.242313+0000","last_change":"2024-04-18T10:05:59.222327+0000","last_active":"2024-04-18T10:06:00.242313+0000","last_peered":"2024-04-18T10:06:00.242313+0000","last_clean":"2024-04-18T10:06:00.242313+0000","last_became_active":"2024-04-18T10:05:59.221623+0000","last_became_peered":"2024-04-18T10:05:59.221623+0000","last_unstale":"2024-04-18T10:06:00.242313+0000","last_undegraded":"2024-04-18T10:06:00.242313+0000","last_fullsized":"2024-04-18T10:06:00.242313+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:58:16.367066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.239652+0000","last_change":"2024-04-18T10:05:58.255714+0000","last_active":"2024-04-18T10:06:00.239652+0000","last_peered":"2024-04-18T10:06:00.239652+0000","last_clean":"2024-04-18T10:06:00.239652+0000","last_became_active":"2024-04-18T10:05:58.255115+0000","last_became_peered":"2024-04-18T10:05:58.255115+0000","last_unstale":"2024-04-18T10:06:00.239652+0000","last_undegraded":"2024-04-18T10:06:00.239652+0000","last_fullsized":"2024-04-18T10:06:00.239652+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:43:50.408798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.240843+0000","last_change":"2024-04-18T10:05:59.244685+0000","last_active":"2024-04-18T10:06:00.240843+0000","last_peered":"2024-04-18T10:06:00.240843+0000","last_clean":"2024-04-18T10:06:00.240843+0000","last_became_active":"2024-04-18T10:05:59.243811+0000","last_became_peered":"2024-04-18T10:05:59.243811+0000","last_unstale":"2024-04-18T10:06:00.240843+0000","last_undegraded":"2024-04-18T10:06:00.240843+0000","last_fullsized":"2024-04-18T10:06:00.240843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:44:42.861177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.241014+0000","last_change":"2024-04-18T10:05:58.256922+0000","last_active":"2024-04-18T10:06:00.241014+0000","last_peered":"2024-04-18T10:06:00.241014+0000","last_clean":"2024-04-18T10:06:00.241014+0000","last_became_active":"2024-04-18T10:05:58.255255+0000","last_became_peered":"2024-04-18T10:05:58.255255+0000","last_unstale":"2024-04-18T10:06:00.241014+0000","last_undegraded":"2024-04-18T10:06:00.241014+0000","last_fullsized":"2024-04-18T10:06:00.241014+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:04:26.651051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383632+0000","last_change":"2024-04-18T10:05:59.253025+0000","last_active":"2024-04-18T10:06:00.383632+0000","last_peered":"2024-04-18T10:06:00.383632+0000","last_clean":"2024-04-18T10:06:00.383632+0000","last_became_active":"2024-04-18T10:05:59.252708+0000","last_became_peered":"2024-04-18T10:05:59.252708+0000","last_unstale":"2024-04-18T10:06:00.383632+0000","last_undegraded":"2024-04-18T10:06:00.383632+0000","last_fullsized":"2024-04-18T10:06:00.383632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:17:23.117921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386955+0000","last_change":"2024-04-18T10:05:58.234122+0000","last_active":"2024-04-18T10:06:00.386955+0000","last_peered":"2024-04-18T10:06:00.386955+0000","last_clean":"2024-04-18T10:06:00.386955+0000","last_became_active":"2024-04-18T10:05:58.233320+0000","last_became_peered":"2024-04-18T10:05:58.233320+0000","last_unstale":"2024-04-18T10:06:00.386955+0000","last_undegraded":"2024-04-18T10:06:00.386955+0000","last_fullsized":"2024-04-18T10:06:00.386955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:54:27.239672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.386581+0000","last_change":"2024-04-18T10:05:59.244230+0000","last_active":"2024-04-18T10:06:00.386581+0000","last_peered":"2024-04-18T10:06:00.386581+0000","last_clean":"2024-04-18T10:06:00.386581+0000","last_became_active":"2024-04-18T10:05:59.243879+0000","last_became_peered":"2024-04-18T10:05:59.243879+0000","last_unstale":"2024-04-18T10:06:00.386581+0000","last_undegraded":"2024-04-18T10:06:00.386581+0000","last_fullsized":"2024-04-18T10:06:00.386581+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:58.190358+0000","last_clean_scrub_stamp":"2024-04-18T10:05:58.190358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:34:28.871228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-18T10:06:00.383948+0000","last_change":"2024-04-18T10:05:58.247996+0000","last_active":"2024-04-18T10:06:00.383948+0000","last_peered":"2024-04-18T10:06:00.383948+0000","last_clean":"2024-04-18T10:06:00.383948+0000","last_became_active":"2024-04-18T10:05:58.247612+0000","last_became_peered":"2024-04-18T10:05:58.247612+0000","last_unstale":"2024-04-18T10:06:00.383948+0000","last_undegraded":"2024-04-18T10:06:00.383948+0000","last_fullsized":"2024-04-18T10:06:00.383948+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:57.179323+0000","last_clean_scrub_stamp":"2024-04-18T10:05:57.179323+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:13:49.398609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":75695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":229376,"data_stored":77217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":823296,"data_stored":666611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T10:06:06.444 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-18T10:06:06.444 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-18T10:06:06.445 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-18T10:06:06.445 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-18T10:06:06.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744c752640 1 Processor -- start 2024-04-18T10:06:06.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744c752640 1 -- start start 2024-04-18T10:06:06.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744c752640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74440b9d90 0x7f74441450e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744c752640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f74440b4940 0x7f7444148840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744c752640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7444145620 con 0x7f74440b44e0 2024-04-18T10:06:06.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744c752640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f74440b4dd0 con 0x7f74440b4940 2024-04-18T10:06:06.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744c752640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7444145b80 con 0x7f74440b9d90 2024-04-18T10:06:06.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744acef640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f74440b4940 0x7f7444148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744acef640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f74440b4940 0x7f7444148840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53430/0 (socket says 172.21.15.62:53430) 2024-04-18T10:06:06.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744acef640 1 -- 172.21.15.62:0/3330591742 learned_addr learned my addr 172.21.15.62:0/3330591742 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:06.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f7449ced640 1 --1- 172.21.15.62:0/3330591742 >> v1:172.21.15.62:6789/0 conn(0x7f74440b44e0 0x7f7444148130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55000/0 (socket says 172.21.15.62:55000) 2024-04-18T10:06:06.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744acef640 1 -- 172.21.15.62:0/3330591742 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74440b9d90 msgr2=0x7f74441450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:06.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744acef640 1 --2- 172.21.15.62:0/3330591742 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74440b9d90 0x7f74441450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744acef640 1 -- 172.21.15.62:0/3330591742 >> v1:172.21.15.62:6789/0 conn(0x7f74440b44e0 legacy=0x7f7444148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.606+0000 7f744acef640 1 -- 172.21.15.62:0/3330591742 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74440ba170 con 0x7f74440b4940 2024-04-18T10:06:06.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f744acef640 1 --2- 172.21.15.62:0/3330591742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74440b4940 0x7f7444148840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f74340098a0 tx=0x7f743401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306540304d1e1476 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:06.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f74494ec640 1 -- 172.21.15.62:0/3330591742 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74340b0070 con 0x7f74440b4940 2024-04-18T10:06:06.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f74494ec640 1 -- 172.21.15.62:0/3330591742 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74340ab600 con 0x7f74440b4940 2024-04-18T10:06:06.609 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f74494ec640 1 -- 172.21.15.62:0/3330591742 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74340ab9d0 con 0x7f74440b4940 2024-04-18T10:06:06.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f744c752640 1 -- 172.21.15.62:0/3330591742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74440b4940 msgr2=0x7f7444148840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f744c752640 1 --2- 172.21.15.62:0/3330591742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74440b4940 0x7f7444148840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f74340098a0 tx=0x7f743401c4b0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f744c752640 1 -- 172.21.15.62:0/3330591742 shutdown_connections 2024-04-18T10:06:06.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f744c752640 1 --2- 172.21.15.62:0/3330591742 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74440b4940 0x7f7444148840 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f744c752640 1 --2- 172.21.15.62:0/3330591742 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74440b9d90 0x7f74441450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f744c752640 1 -- 172.21.15.62:0/3330591742 >> 172.21.15.62:0/3330591742 conn(0x7f744406c0d0 msgr2=0x7f74440b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:06.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f744c752640 1 -- 172.21.15.62:0/3330591742 shutdown_connections 2024-04-18T10:06:06.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f744c752640 1 -- 172.21.15.62:0/3330591742 wait complete. 2024-04-18T10:06:06.612 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f744c752640 1 Processor -- start 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.610+0000 7f744c752640 1 -- start start 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744c752640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74440b44e0 0x7f74440b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744c752640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f74440b4940 0x7f74441e0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744c752640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74440b9d90 0x7f74441e0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744c752640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f744409e3d0 con 0x7f74440b9d90 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744c752640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7444148d80 con 0x7f74440b4940 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744c752640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f74440b82c0 con 0x7f74440b44e0 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744acef640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74440b9d90 0x7f74441e0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744acef640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74440b9d90 0x7f74441e0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53436/0 (socket says 172.21.15.62:53436) 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744acef640 1 -- 172.21.15.62:0/3727070549 learned_addr learned my addr 172.21.15.62:0/3727070549 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f7449ced640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f74440b4940 0x7f74441e0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744a4ee640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74440b44e0 0x7f74440b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744acef640 1 -- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74440b44e0 msgr2=0x7f74440b13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744acef640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74440b44e0 0x7f74440b13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744acef640 1 -- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f74440b4940 msgr2=0x7f74441e0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744acef640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f74440b4940 0x7f74441e0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744acef640 1 -- 172.21.15.62:0/3727070549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744408c050 con 0x7f74440b9d90 2024-04-18T10:06:06.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744a4ee640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74440b44e0 0x7f74440b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:06.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744acef640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74440b9d90 0x7f74441e0860 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f74340098a0 tx=0x7f743401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:06.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f7442ffd640 1 -- 172.21.15.62:0/3727070549 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74340ba020 con 0x7f74440b9d90 2024-04-18T10:06:06.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f7442ffd640 1 -- 172.21.15.62:0/3727070549 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74340b6620 con 0x7f74440b9d90 2024-04-18T10:06:06.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f7442ffd640 1 -- 172.21.15.62:0/3727070549 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74340b0040 con 0x7f74440b9d90 2024-04-18T10:06:06.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744c752640 1 -- 172.21.15.62:0/3727070549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f744414efa0 con 0x7f74440b9d90 2024-04-18T10:06:06.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744c752640 1 -- 172.21.15.62:0/3727070549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f744408c2c0 con 0x7f74440b9d90 2024-04-18T10:06:06.614 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744c752640 1 -- 172.21.15.62:0/3727070549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74440b8440 con 0x7f74440b9d90 2024-04-18T10:06:06.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f7442ffd640 1 -- 172.21.15.62:0/3727070549 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f74340bd020 con 0x7f74440b9d90 2024-04-18T10:06:06.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f7442ffd640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7430068380 0x7f743006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f7442ffd640 1 -- 172.21.15.62:0/3727070549 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7434120c60 con 0x7f74440b9d90 2024-04-18T10:06:06.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.614+0000 7f744a4ee640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7430068380 0x7f743006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.618+0000 7f744a4ee640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7430068380 0x7f743006a840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f743c007eb0 tx=0x7f743c0073c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:06.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.618+0000 7f7442ffd640 1 -- 172.21.15.62:0/3727070549 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f74340c4030 con 0x7f74440b9d90 2024-04-18T10:06:06.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.814+0000 7f744c752640 1 -- 172.21.15.62:0/3727070549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f7444068b90 con 0x7f74440b9d90 2024-04-18T10:06:06.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.814+0000 7f7442ffd640 1 -- 172.21.15.62:0/3727070549 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f7434016130 con 0x7f74440b9d90 2024-04-18T10:06:06.814 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:06.814 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-18T10:06:06.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.818+0000 7f744c752640 1 -- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7430068380 msgr2=0x7f743006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.818+0000 7f744c752640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7430068380 0x7f743006a840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f743c007eb0 tx=0x7f743c0073c0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.818+0000 7f744c752640 1 -- 172.21.15.62:0/3727070549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74440b9d90 msgr2=0x7f74441e0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.818+0000 7f744c752640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74440b9d90 0x7f74441e0860 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f74340098a0 tx=0x7f743401acf0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.818+0000 7f744c752640 1 -- 172.21.15.62:0/3727070549 shutdown_connections 2024-04-18T10:06:06.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.818+0000 7f744c752640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7430068380 0x7f743006a840 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.818+0000 7f744c752640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f74440b9d90 0x7f74441e0860 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.818+0000 7f744c752640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f74440b4940 0x7f74441e0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.818+0000 7f744c752640 1 --2- 172.21.15.62:0/3727070549 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f74440b44e0 0x7f74440b13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.818+0000 7f744c752640 1 -- 172.21.15.62:0/3727070549 >> 172.21.15.62:0/3727070549 conn(0x7f744406c0d0 msgr2=0x7f74440b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:06.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.818+0000 7f744c752640 1 -- 172.21.15.62:0/3727070549 shutdown_connections 2024-04-18T10:06:06.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.818+0000 7f744c752640 1 -- 172.21.15.62:0/3727070549 wait complete. 2024-04-18T10:06:06.832 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-18T10:06:06.832 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:06.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bcb4bb640 1 Processor -- start 2024-04-18T10:06:06.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bcb4bb640 1 -- start start 2024-04-18T10:06:06.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bcb4bb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bc40b6460 0x7f5bc40b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bcb4bb640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5bc40b8000 0x7f5bc41418c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bcb4bb640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5bc4141e00 0x7f5bc414b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bcb4bb640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc40b9f40 con 0x7f5bc4141e00 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bcb4bb640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f5bc40b6840 con 0x7f5bc40b8000 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bcb4bb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc4147fb0 con 0x7f5bc40b6460 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bc9a58640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5bc4141e00 0x7f5bc414b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bc9a58640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5bc4141e00 0x7f5bc414b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53438/0 (socket says 172.21.15.62:53438) 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bc9a58640 1 -- 172.21.15.62:0/4239088712 learned_addr learned my addr 172.21.15.62:0/4239088712 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bc8a56640 1 --2- 172.21.15.62:0/4239088712 >> v2:172.21.15.145:3300/0 conn(0x7f5bc40b8000 0x7f5bc41418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bc9257640 1 --2- 172.21.15.62:0/4239088712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bc40b6460 0x7f5bc40b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bc9a58640 1 -- 172.21.15.62:0/4239088712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bc40b6460 msgr2=0x7f5bc40b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bc9a58640 1 --2- 172.21.15.62:0/4239088712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bc40b6460 0x7f5bc40b5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bc9a58640 1 -- 172.21.15.62:0/4239088712 >> v2:172.21.15.145:3300/0 conn(0x7f5bc40b8000 msgr2=0x7f5bc41418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bc9a58640 1 --2- 172.21.15.62:0/4239088712 >> v2:172.21.15.145:3300/0 conn(0x7f5bc40b8000 0x7f5bc41418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bc9a58640 1 -- 172.21.15.62:0/4239088712 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc4148290 con 0x7f5bc4141e00 2024-04-18T10:06:06.954 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bc8a56640 1 --2- 172.21.15.62:0/4239088712 >> v2:172.21.15.145:3300/0 conn(0x7f5bc40b8000 0x7f5bc41418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:06.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.954+0000 7f5bc9257640 1 --2- 172.21.15.62:0/4239088712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bc40b6460 0x7f5bc40b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:06.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bc9a58640 1 --2- 172.21.15.62:0/4239088712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc4141e00 0x7f5bc414b290 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5bc406c450 tx=0x7f5bc001c1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f19f2d504e0eee server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:06.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bbbfff640 1 -- 172.21.15.62:0/4239088712 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00b0070 con 0x7f5bc4141e00 2024-04-18T10:06:06.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bbbfff640 1 -- 172.21.15.62:0/4239088712 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bc00ab600 con 0x7f5bc4141e00 2024-04-18T10:06:06.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bbbfff640 1 -- 172.21.15.62:0/4239088712 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bc00ab9d0 con 0x7f5bc4141e00 2024-04-18T10:06:06.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/4239088712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc4141e00 msgr2=0x7f5bc414b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 --2- 172.21.15.62:0/4239088712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc4141e00 0x7f5bc414b290 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5bc406c450 tx=0x7f5bc001c1a0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/4239088712 shutdown_connections 2024-04-18T10:06:06.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 --2- 172.21.15.62:0/4239088712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc4141e00 0x7f5bc414b290 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 --2- 172.21.15.62:0/4239088712 >> v2:172.21.15.145:3300/0 conn(0x7f5bc40b8000 0x7f5bc41418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 --2- 172.21.15.62:0/4239088712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bc40b6460 0x7f5bc40b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/4239088712 >> 172.21.15.62:0/4239088712 conn(0x7f5bc40bd600 msgr2=0x7f5bc40bda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:06.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/4239088712 shutdown_connections 2024-04-18T10:06:06.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/4239088712 wait complete. 2024-04-18T10:06:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 Processor -- start 2024-04-18T10:06:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 -- start start 2024-04-18T10:06:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc40b6460 0x7f5bc414c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bc40b8000 0x7f5bc414a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bc4141e00 0x7f5bc41e3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc41423f0 con 0x7f5bc40b6460 2024-04-18T10:06:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc4142270 con 0x7f5bc40b8000 2024-04-18T10:06:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f5bc4148a90 con 0x7f5bc4141e00 2024-04-18T10:06:06.959 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bc9257640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc40b6460 0x7f5bc414c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bc9257640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc40b6460 0x7f5bc414c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53446/0 (socket says 172.21.15.62:53446) 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bc9257640 1 -- 172.21.15.62:0/1572881187 learned_addr learned my addr 172.21.15.62:0/1572881187 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bc8a56640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bc40b8000 0x7f5bc414a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bc9a58640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bc4141e00 0x7f5bc41e3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bc9257640 1 -- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bc4141e00 msgr2=0x7f5bc41e3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bc9257640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bc4141e00 0x7f5bc41e3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bc9257640 1 -- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bc40b8000 msgr2=0x7f5bc414a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bc9257640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bc40b8000 0x7f5bc414a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bc9257640 1 -- 172.21.15.62:0/1572881187 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bc414a9f0 con 0x7f5bc40b6460 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bc9257640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc40b6460 0x7f5bc414c3a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc00cf90 tx=0x7f5bbc00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/1572881187 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc01a070 con 0x7f5bc40b6460 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/1572881187 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bbc00d840 con 0x7f5bc40b6460 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/1572881187 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bc414ca00 con 0x7f5bc40b6460 2024-04-18T10:06:06.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/1572881187 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc00dc10 con 0x7f5bc40b6460 2024-04-18T10:06:06.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.958+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/1572881187 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bc41e34c0 con 0x7f5bc40b6460 2024-04-18T10:06:06.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.962+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/1572881187 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bc4068b90 con 0x7f5bc40b6460 2024-04-18T10:06:06.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.962+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/1572881187 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5bbc01d070 con 0x7f5bc40b6460 2024-04-18T10:06:06.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.962+0000 7f5bb9ffb640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5ba8068380 0x7f5ba806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:06.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.962+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/1572881187 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5bbc087090 con 0x7f5bc40b6460 2024-04-18T10:06:06.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.962+0000 7f5bc8a56640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5ba8068380 0x7f5ba806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:06.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.962+0000 7f5bc8a56640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5ba8068380 0x7f5ba806a840 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4006fd0 tx=0x7f5bb4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:06.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:06.966+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/1572881187 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5bbc057090 con 0x7f5bc40b6460 2024-04-18T10:06:07.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.122+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/1572881187 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5bc40b53d0 con 0x7f5bc40b6460 2024-04-18T10:06:07.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.122+0000 7f5bb9ffb640 1 -- 172.21.15.62:0/1572881187 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f5bbc05b390 con 0x7f5bc40b6460 2024-04-18T10:06:07.123 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:07.123 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:07.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.126+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5ba8068380 msgr2=0x7f5ba806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.126+0000 7f5bcb4bb640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5ba8068380 0x7f5ba806a840 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5bb4006fd0 tx=0x7f5bb4008040 comp rx=0 tx=0).stop 2024-04-18T10:06:07.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.126+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/1572881187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc40b6460 msgr2=0x7f5bc414c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.126+0000 7f5bcb4bb640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc40b6460 0x7f5bc414c3a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc00cf90 tx=0x7f5bbc00a7d0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.126+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/1572881187 shutdown_connections 2024-04-18T10:06:07.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.126+0000 7f5bcb4bb640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5ba8068380 0x7f5ba806a840 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.126+0000 7f5bcb4bb640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5bc4141e00 0x7f5bc41e3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.126+0000 7f5bcb4bb640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5bc40b8000 0x7f5bc414a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.126+0000 7f5bcb4bb640 1 --2- 172.21.15.62:0/1572881187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5bc40b6460 0x7f5bc414c3a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.126+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/1572881187 >> 172.21.15.62:0/1572881187 conn(0x7f5bc40bd600 msgr2=0x7f5bc40b85f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:07.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.126+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/1572881187 shutdown_connections 2024-04-18T10:06:07.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.126+0000 7f5bcb4bb640 1 -- 172.21.15.62:0/1572881187 wait complete. 2024-04-18T10:06:07.140 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-18T10:05:56.782649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-18T10:05:57.541623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:07.141 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:07.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f477f1d9640 1 Processor -- start 2024-04-18T10:06:07.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f477f1d9640 1 -- start start 2024-04-18T10:06:07.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f477f1d9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f477809e550 0x7f477809f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f477f1d9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f477806c110 con 0x7f4778149390 2024-04-18T10:06:07.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f477f1d9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47780b62f0 con 0x7f477806bd30 2024-04-18T10:06:07.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f477f1d9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f477809de90 con 0x7f477809e550 2024-04-18T10:06:07.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f4777fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f477809e550 0x7f477809f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f477d776640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4778149390 0x7f47781497e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55004/0 (socket says 172.21.15.62:55004) 2024-04-18T10:06:07.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f4777fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f477809e550 0x7f477809f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53462/0 (socket says 172.21.15.62:53462) 2024-04-18T10:06:07.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f477d776640 1 -- 172.21.15.62:0/2495485272 learned_addr learned my addr 172.21.15.62:0/2495485272 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:07.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f4777fff640 1 -- 172.21.15.62:0/2495485272 >> v1:172.21.15.145:6789/0 conn(0x7f477806bd30 legacy=0x7f47780b5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:07.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f4777fff640 1 -- 172.21.15.62:0/2495485272 >> v1:172.21.15.62:6789/0 conn(0x7f4778149390 legacy=0x7f47781497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.266+0000 7f4777fff640 1 -- 172.21.15.62:0/2495485272 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47780b65d0 con 0x7f477809e550 2024-04-18T10:06:07.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f4777fff640 1 --2- 172.21.15.62:0/2495485272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f477809e550 0x7f477809f320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f476c0097c0 tx=0x7f476c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b22d172093e114d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:07.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f47777fe640 1 -- 172.21.15.62:0/2495485272 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c0b0070 con 0x7f477809e550 2024-04-18T10:06:07.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f47777fe640 1 -- 172.21.15.62:0/2495485272 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476c0ab600 con 0x7f477809e550 2024-04-18T10:06:07.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f47777fe640 1 -- 172.21.15.62:0/2495485272 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c0ab9d0 con 0x7f477809e550 2024-04-18T10:06:07.268 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f477f1d9640 1 -- 172.21.15.62:0/2495485272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f477809e550 msgr2=0x7f477809f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f477f1d9640 1 --2- 172.21.15.62:0/2495485272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f477809e550 0x7f477809f320 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f476c0097c0 tx=0x7f476c01c3d0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f477f1d9640 1 -- 172.21.15.62:0/2495485272 shutdown_connections 2024-04-18T10:06:07.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f477f1d9640 1 --2- 172.21.15.62:0/2495485272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f477809e550 0x7f477809f320 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f477f1d9640 1 -- 172.21.15.62:0/2495485272 >> 172.21.15.62:0/2495485272 conn(0x7f47780ba710 msgr2=0x7f47780bab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:07.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f477f1d9640 1 -- 172.21.15.62:0/2495485272 shutdown_connections 2024-04-18T10:06:07.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f477f1d9640 1 -- 172.21.15.62:0/2495485272 wait complete. 2024-04-18T10:06:07.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f477f1d9640 1 Processor -- start 2024-04-18T10:06:07.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.270+0000 7f477f1d9640 1 -- start start 2024-04-18T10:06:07.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f477f1d9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f477806bd30 0x7f477809c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f477f1d9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f477809e550 0x7f47780b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f477f1d9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4778149390 0x7f4778143640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f477f1d9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f47781548a0 con 0x7f477809e550 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f477f1d9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f4778154720 con 0x7f477806bd30 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f477f1d9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f4778154a20 con 0x7f4778149390 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f4777fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f477809e550 0x7f47780b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f4777fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f477809e550 0x7f47780b2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53464/0 (socket says 172.21.15.62:53464) 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f4777fff640 1 -- 172.21.15.62:0/3684446649 learned_addr learned my addr 172.21.15.62:0/3684446649 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f477d776640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4778149390 0x7f4778143640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f4777fff640 1 -- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4778149390 msgr2=0x7f4778143640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f4777fff640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4778149390 0x7f4778143640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f4777fff640 1 -- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f477806bd30 msgr2=0x7f477809c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f4777fff640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f477806bd30 0x7f477809c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f4777fff640 1 -- 172.21.15.62:0/3684446649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47780b36a0 con 0x7f477809e550 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f4777fff640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f477809e550 0x7f47780b2f20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f476c0ae040 tx=0x7f476c01ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:07.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f47757fa640 1 -- 172.21.15.62:0/3684446649 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c0b0050 con 0x7f477809e550 2024-04-18T10:06:07.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f47757fa640 1 -- 172.21.15.62:0/3684446649 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476c0bc070 con 0x7f477809e550 2024-04-18T10:06:07.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f47757fa640 1 -- 172.21.15.62:0/3684446649 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c0b5870 con 0x7f477809e550 2024-04-18T10:06:07.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f477f1d9640 1 -- 172.21.15.62:0/3684446649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47780b3980 con 0x7f477809e550 2024-04-18T10:06:07.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f477f1d9640 1 -- 172.21.15.62:0/3684446649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47780b1d50 con 0x7f477809e550 2024-04-18T10:06:07.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f477f1d9640 1 -- 172.21.15.62:0/3684446649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4778069230 con 0x7f477809e550 2024-04-18T10:06:07.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f47757fa640 1 -- 172.21.15.62:0/3684446649 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f476c0b5a10 con 0x7f477809e550 2024-04-18T10:06:07.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f47757fa640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f475c068380 0x7f475c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f47757fa640 1 -- 172.21.15.62:0/3684446649 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f476c1211e0 con 0x7f477809e550 2024-04-18T10:06:07.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.274+0000 7f477cf75640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f475c068380 0x7f475c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.278+0000 7f477cf75640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f475c068380 0x7f475c06a840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f477809cc40 tx=0x7f4764006bd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:07.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.278+0000 7f47757fa640 1 -- 172.21.15.62:0/3684446649 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f476c016050 con 0x7f477809e550 2024-04-18T10:06:07.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.434+0000 7f477f1d9640 1 -- 172.21.15.62:0/3684446649 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f477813e0c0 con 0x7f477809e550 2024-04-18T10:06:07.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f47757fa640 1 -- 172.21.15.62:0/3684446649 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f476c0f1830 con 0x7f477809e550 2024-04-18T10:06:07.436 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:07.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f477f1d9640 1 -- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f475c068380 msgr2=0x7f475c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f477f1d9640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f475c068380 0x7f475c06a840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f477809cc40 tx=0x7f4764006bd0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f477f1d9640 1 -- 172.21.15.62:0/3684446649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f477809e550 msgr2=0x7f47780b2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f477f1d9640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f477809e550 0x7f47780b2f20 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f476c0ae040 tx=0x7f476c01ac10 comp rx=0 tx=0).stop 2024-04-18T10:06:07.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f477f1d9640 1 -- 172.21.15.62:0/3684446649 shutdown_connections 2024-04-18T10:06:07.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f477f1d9640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f475c068380 0x7f475c06a840 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f477f1d9640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4778149390 0x7f4778143640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f477f1d9640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f477809e550 0x7f47780b2f20 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f477f1d9640 1 --2- 172.21.15.62:0/3684446649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f477806bd30 0x7f477809c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f477f1d9640 1 -- 172.21.15.62:0/3684446649 >> 172.21.15.62:0/3684446649 conn(0x7f47780ba710 msgr2=0x7f47780b96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:07.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f477f1d9640 1 -- 172.21.15.62:0/3684446649 shutdown_connections 2024-04-18T10:06:07.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.438+0000 7f477f1d9640 1 -- 172.21.15.62:0/3684446649 wait complete. 2024-04-18T10:06:07.453 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-18T10:06:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.578+0000 7f7087241640 1 Processor -- start 2024-04-18T10:06:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f7087241640 1 -- start start 2024-04-18T10:06:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f7087241640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70800ae4e0 0x7f708013e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f7087241640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f70800737c0 0x7f708013eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f7087241640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70800b0580 con 0x7f70800b01a0 2024-04-18T10:06:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f7087241640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f70800b59d0 con 0x7f70800737c0 2024-04-18T10:06:07.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f7087241640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f70800ae970 con 0x7f70800ae4e0 2024-04-18T10:06:07.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f707ffff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70800ae4e0 0x7f708013e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f70857de640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f70800737c0 0x7f708013eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f7084fdd640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f70800b01a0 0x7f708006d2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.62:53706/0 (socket says 172.21.15.62:53706) 2024-04-18T10:06:07.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f7084fdd640 1 -- 172.21.15.62:0/3344523858 learned_addr learned my addr 172.21.15.62:0/3344523858 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:07.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f707ffff640 1 -- 172.21.15.62:0/3344523858 >> v2:172.21.15.145:3300/0 conn(0x7f70800737c0 msgr2=0x7f708013eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:07.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f707ffff640 1 --2- 172.21.15.62:0/3344523858 >> v2:172.21.15.145:3300/0 conn(0x7f70800737c0 0x7f708013eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f707ffff640 1 -- 172.21.15.62:0/3344523858 >> v1:172.21.15.145:6789/0 conn(0x7f70800b01a0 legacy=0x7f708006d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f707ffff640 1 -- 172.21.15.62:0/3344523858 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70800b53a0 con 0x7f70800ae4e0 2024-04-18T10:06:07.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.582+0000 7f70857de640 1 --2- 172.21.15.62:0/3344523858 >> v2:172.21.15.145:3300/0 conn(0x7f70800737c0 0x7f708013eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:06:07.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f707ffff640 1 --2- 172.21.15.62:0/3344523858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70800ae4e0 0x7f708013e980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f706c0097c0 tx=0x7f706c01c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=138cd0dfb5975675 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:07.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f707f7fe640 1 -- 172.21.15.62:0/3344523858 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c0b0070 con 0x7f70800ae4e0 2024-04-18T10:06:07.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f707f7fe640 1 -- 172.21.15.62:0/3344523858 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f706c0ab600 con 0x7f70800ae4e0 2024-04-18T10:06:07.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f707f7fe640 1 -- 172.21.15.62:0/3344523858 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c0ab9d0 con 0x7f70800ae4e0 2024-04-18T10:06:07.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 -- 172.21.15.62:0/3344523858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70800ae4e0 msgr2=0x7f708013e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 --2- 172.21.15.62:0/3344523858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70800ae4e0 0x7f708013e980 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f706c0097c0 tx=0x7f706c01c380 comp rx=0 tx=0).stop 2024-04-18T10:06:07.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 -- 172.21.15.62:0/3344523858 shutdown_connections 2024-04-18T10:06:07.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 --2- 172.21.15.62:0/3344523858 >> v2:172.21.15.145:3300/0 conn(0x7f70800737c0 0x7f708013eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 --2- 172.21.15.62:0/3344523858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70800ae4e0 0x7f708013e980 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 -- 172.21.15.62:0/3344523858 >> 172.21.15.62:0/3344523858 conn(0x7f70800b9690 msgr2=0x7f70800b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:07.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 -- 172.21.15.62:0/3344523858 shutdown_connections 2024-04-18T10:06:07.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 -- 172.21.15.62:0/3344523858 wait complete. 2024-04-18T10:06:07.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 Processor -- start 2024-04-18T10:06:07.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 -- start start 2024-04-18T10:06:07.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70800737c0 0x7f70801447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70800b01a0 0x7f7080071be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70801e0440 0x7f70801e4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7080151e60 con 0x7f70800b01a0 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f708006e470 con 0x7f70801e0440 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f7087241640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7080151fe0 con 0x7f70800737c0 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.586+0000 7f707ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70800b01a0 0x7f7080071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70800b01a0 0x7f7080071be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53474/0 (socket says 172.21.15.62:53474) 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707ffff640 1 -- 172.21.15.62:0/1673820964 learned_addr learned my addr 172.21.15.62:0/1673820964 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f7084fdd640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70800737c0 0x7f70801447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f70857de640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70801e0440 0x7f70801e4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707ffff640 1 -- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70800737c0 msgr2=0x7f70801447c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707ffff640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70800737c0 0x7f70801447c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707ffff640 1 -- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70801e0440 msgr2=0x7f70801e4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.589 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707ffff640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70801e0440 0x7f70801e4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707ffff640 1 -- 172.21.15.62:0/1673820964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70801e3a50 con 0x7f70800b01a0 2024-04-18T10:06:07.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707ffff640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70800b01a0 0x7f7080071be0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f706c019fb0 tx=0x7f706c019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:07.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707d7fa640 1 -- 172.21.15.62:0/1673820964 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c0b8020 con 0x7f70800b01a0 2024-04-18T10:06:07.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707d7fa640 1 -- 172.21.15.62:0/1673820964 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f706c0b4410 con 0x7f70800b01a0 2024-04-18T10:06:07.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f7087241640 1 -- 172.21.15.62:0/1673820964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70801e3d30 con 0x7f70800b01a0 2024-04-18T10:06:07.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707d7fa640 1 -- 172.21.15.62:0/1673820964 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f706c0b0040 con 0x7f70800b01a0 2024-04-18T10:06:07.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f7087241640 1 -- 172.21.15.62:0/1673820964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f708009e2d0 con 0x7f70800b01a0 2024-04-18T10:06:07.590 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f7087241640 1 -- 172.21.15.62:0/1673820964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7080068c00 con 0x7f70800b01a0 2024-04-18T10:06:07.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707d7fa640 1 -- 172.21.15.62:0/1673820964 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f706c0bb020 con 0x7f70800b01a0 2024-04-18T10:06:07.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707d7fa640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7064068380 0x7f706406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f707d7fa640 1 -- 172.21.15.62:0/1673820964 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f706c120740 con 0x7f70800b01a0 2024-04-18T10:06:07.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.590+0000 7f7084fdd640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7064068380 0x7f706406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.591 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.594+0000 7f7084fdd640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7064068380 0x7f706406a840 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7074000f70 tx=0x7f7074009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:07.593 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.594+0000 7f707d7fa640 1 -- 172.21.15.62:0/1673820964 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f706c0c2030 con 0x7f70800b01a0 2024-04-18T10:06:07.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f7087241640 1 -- 172.21.15.62:0/1673820964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f708013d400 con 0x7f70800b01a0 2024-04-18T10:06:07.756 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f707d7fa640 1 -- 172.21.15.62:0/1673820964 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f706c016090 con 0x7f70800b01a0 2024-04-18T10:06:07.756 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-18T10:06:07.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f7087241640 1 -- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7064068380 msgr2=0x7f706406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f7087241640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7064068380 0x7f706406a840 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7074000f70 tx=0x7f7074009040 comp rx=0 tx=0).stop 2024-04-18T10:06:07.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f7087241640 1 -- 172.21.15.62:0/1673820964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70800b01a0 msgr2=0x7f7080071be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f7087241640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70800b01a0 0x7f7080071be0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f706c019fb0 tx=0x7f706c019120 comp rx=0 tx=0).stop 2024-04-18T10:06:07.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f7087241640 1 -- 172.21.15.62:0/1673820964 shutdown_connections 2024-04-18T10:06:07.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f7087241640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7064068380 0x7f706406a840 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.759 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f7087241640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f70801e0440 0x7f70801e4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f7087241640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f70800b01a0 0x7f7080071be0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f7087241640 1 --2- 172.21.15.62:0/1673820964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70800737c0 0x7f70801447c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f7087241640 1 -- 172.21.15.62:0/1673820964 >> 172.21.15.62:0/1673820964 conn(0x7f70800b9690 msgr2=0x7f708009aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:07.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.758+0000 7f7087241640 1 -- 172.21.15.62:0/1673820964 shutdown_connections 2024-04-18T10:06:07.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.762+0000 7f7087241640 1 -- 172.21.15.62:0/1673820964 wait complete. 2024-04-18T10:06:07.776 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-18T10:06:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.902+0000 7f0903ad3640 1 Processor -- start 2024-04-18T10:06:07.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.902+0000 7f0903ad3640 1 -- start start 2024-04-18T10:06:07.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f0903ad3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f08fc06e490 0x7f08fc151420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f0903ad3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08fc06d4f0 con 0x7f08fc0b4c90 2024-04-18T10:06:07.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f0903ad3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08fc06d7d0 con 0x7f08fc146690 2024-04-18T10:06:07.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f0903ad3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f08fc06d2a0 con 0x7f08fc06e490 2024-04-18T10:06:07.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090186f640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f08fc0b4c90 0x7f08fc145f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55008/0 (socket says 172.21.15.62:55008) 2024-04-18T10:06:07.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090186f640 1 -- 172.21.15.62:0/1829190829 learned_addr learned my addr 172.21.15.62:0/1829190829 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:07.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f0902070640 1 --2- 172.21.15.62:0/1829190829 >> v2:172.21.15.145:3300/0 conn(0x7f08fc06e490 0x7f08fc151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3293963304 0 0) 0x7f08fc06d4f0 con 0x7f08fc0b4c90 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08e8003530 con 0x7f08fc0b4c90 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 665606376 0 0) 0x7f08e8003530 con 0x7f08fc0b4c90 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 >> v2:172.21.15.145:3300/0 conn(0x7f08fc06e490 msgr2=0x7f08fc151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 --2- 172.21.15.62:0/1829190829 >> v2:172.21.15.145:3300/0 conn(0x7f08fc06e490 0x7f08fc151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 >> v1:172.21.15.145:6789/0 conn(0x7f08fc146690 legacy=0x7f08fc09e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08fc09c570 con 0x7f08fc0b4c90 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f08f8001680 con 0x7f08fc0b4c90 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 >> v1:172.21.15.62:6789/0 conn(0x7f08fc0b4c90 legacy=0x7f08fc145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e80039c0 0x7f08e8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08e8004810 0x7f08e8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e80051d0 0x7f08e800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc06d2a0 con 0x7f08e8004810 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f08e800c170 con 0x7f08e80039c0 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f08e800c370 con 0x7f08e80051d0 2024-04-18T10:06:07.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090106e640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08e8004810 0x7f08e8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090186f640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e80039c0 0x7f08e8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f0902070640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e80051d0 0x7f08e800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090106e640 1 -- 172.21.15.62:0/1829190829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e80051d0 msgr2=0x7f08e800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:07.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090106e640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e80051d0 0x7f08e800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090106e640 1 -- 172.21.15.62:0/1829190829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e80039c0 msgr2=0x7f08e8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:07.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090106e640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e80039c0 0x7f08e8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090106e640 1 -- 172.21.15.62:0/1829190829 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08e8003530 con 0x7f08e8004810 2024-04-18T10:06:07.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f090186f640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e80039c0 0x7f08e8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:07.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f0902070640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e80051d0 0x7f08e800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:07.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.906+0000 7f0902070640 1 -- 172.21.15.62:0/1829190829 reap_dead start 2024-04-18T10:06:07.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090106e640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08e8004810 0x7f08e8004c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f08ec001780 tx=0x7f08ec01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb664d16252213a server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:07.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec0013a0 con 0x7f08e8004810 2024-04-18T10:06:07.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08ec0a4d40 con 0x7f08e8004810 2024-04-18T10:06:07.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090086d640 1 -- 172.21.15.62:0/1829190829 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec005310 con 0x7f08e8004810 2024-04-18T10:06:07.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 -- 172.21.15.62:0/1829190829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08e8004810 msgr2=0x7f08e8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08e8004810 0x7f08e8004c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f08ec001780 tx=0x7f08ec01c400 comp rx=0 tx=0).stop 2024-04-18T10:06:07.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 -- 172.21.15.62:0/1829190829 shutdown_connections 2024-04-18T10:06:07.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 --2- 172.21.15.62:0/1829190829 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08e8004810 0x7f08e8004c90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 -- 172.21.15.62:0/1829190829 >> 172.21.15.62:0/1829190829 conn(0x7f08fc0b9690 msgr2=0x7f08fc0b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 -- 172.21.15.62:0/1829190829 shutdown_connections 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 -- 172.21.15.62:0/1829190829 wait complete. 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 Processor -- start 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 -- start start 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e8004810 0x7f08fc1e2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08fc142400 0x7f08fc070690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08fc070bd0 0x7f08fc13d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc149220 con 0x7f08fc142400 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f08fc1490a0 con 0x7f08e8004810 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f08fc1493a0 con 0x7f08fc070bd0 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090106e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08fc142400 0x7f08fc070690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090106e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08fc142400 0x7f08fc070690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53500/0 (socket says 172.21.15.62:53500) 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090106e640 1 -- 172.21.15.62:0/315018048 learned_addr learned my addr 172.21.15.62:0/315018048 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0902070640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08fc070bd0 0x7f08fc13d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090186f640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e8004810 0x7f08fc1e2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090106e640 1 -- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08fc070bd0 msgr2=0x7f08fc13d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090106e640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08fc070bd0 0x7f08fc13d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090106e640 1 -- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e8004810 msgr2=0x7f08fc1e2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:07.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090106e640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e8004810 0x7f08fc1e2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:07.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090106e640 1 -- 172.21.15.62:0/315018048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08fc1e2c40 con 0x7f08fc142400 2024-04-18T10:06:07.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f090106e640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08fc142400 0x7f08fc070690 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f08ec01c230 tx=0x7f08e8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:07.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f08f27fc640 1 -- 172.21.15.62:0/315018048 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec007a40 con 0x7f08fc142400 2024-04-18T10:06:07.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f08f27fc640 1 -- 172.21.15.62:0/315018048 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08ec0a4b20 con 0x7f08fc142400 2024-04-18T10:06:07.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f08f27fc640 1 -- 172.21.15.62:0/315018048 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ec007240 con 0x7f08fc142400 2024-04-18T10:06:07.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 -- 172.21.15.62:0/315018048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08fc071050 con 0x7f08fc142400 2024-04-18T10:06:07.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.910+0000 7f0903ad3640 1 -- 172.21.15.62:0/315018048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08fc142c30 con 0x7f08fc142400 2024-04-18T10:06:07.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.914+0000 7f0903ad3640 1 -- 172.21.15.62:0/315018048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08fc0704c0 con 0x7f08fc142400 2024-04-18T10:06:07.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.914+0000 7f08f27fc640 1 -- 172.21.15.62:0/315018048 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f08ec0bc060 con 0x7f08fc142400 2024-04-18T10:06:07.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.914+0000 7f08f27fc640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f08e0068380 0x7f08e006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:07.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.914+0000 7f08f27fc640 1 -- 172.21.15.62:0/315018048 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f08ec11f7a0 con 0x7f08fc142400 2024-04-18T10:06:07.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.914+0000 7f090186f640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f08e0068380 0x7f08e006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:07.914 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.914+0000 7f090186f640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f08e0068380 0x7f08e006a840 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f08fc1d51f0 tx=0x7f08f8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:07.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:07.918+0000 7f08f27fc640 1 -- 172.21.15.62:0/315018048 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f08fc071050 con 0x7f08fc142400 2024-04-18T10:06:08.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.074+0000 7f0903ad3640 1 -- 172.21.15.62:0/315018048 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f08fc068c00 con 0x7f08fc142400 2024-04-18T10:06:08.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.074+0000 7f08f27fc640 1 -- 172.21.15.62:0/315018048 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f08fc1e2c40 con 0x7f08fc142400 2024-04-18T10:06:08.074 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-04-18T10:06:08.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.078+0000 7f0903ad3640 1 -- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f08e0068380 msgr2=0x7f08e006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.078+0000 7f0903ad3640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f08e0068380 0x7f08e006a840 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f08fc1d51f0 tx=0x7f08f8009040 comp rx=0 tx=0).stop 2024-04-18T10:06:08.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.078+0000 7f0903ad3640 1 -- 172.21.15.62:0/315018048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08fc142400 msgr2=0x7f08fc070690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.078+0000 7f0903ad3640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08fc142400 0x7f08fc070690 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f08ec01c230 tx=0x7f08e8005840 comp rx=0 tx=0).stop 2024-04-18T10:06:08.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.078+0000 7f0903ad3640 1 -- 172.21.15.62:0/315018048 shutdown_connections 2024-04-18T10:06:08.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.078+0000 7f0903ad3640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f08e0068380 0x7f08e006a840 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.078+0000 7f0903ad3640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08fc070bd0 0x7f08fc13d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.078+0000 7f0903ad3640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f08fc142400 0x7f08fc070690 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.078+0000 7f0903ad3640 1 --2- 172.21.15.62:0/315018048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e8004810 0x7f08fc1e2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.078+0000 7f0903ad3640 1 -- 172.21.15.62:0/315018048 >> 172.21.15.62:0/315018048 conn(0x7f08fc0b9690 msgr2=0x7f08fc09d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:08.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.078+0000 7f0903ad3640 1 -- 172.21.15.62:0/315018048 shutdown_connections 2024-04-18T10:06:08.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.078+0000 7f0903ad3640 1 -- 172.21.15.62:0/315018048 wait complete. 2024-04-18T10:06:08.091 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:08.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.214+0000 7f542cc5f640 1 Processor -- start 2024-04-18T10:06:08.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f542cc5f640 1 -- start start 2024-04-18T10:06:08.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f542cc5f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54280b6260 0x7f5428148260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f542cc5f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f54280b6640 0x7f54280b5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f542cc5f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54280b7850 con 0x7f54281487a0 2024-04-18T10:06:08.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f542cc5f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f5428148cb0 con 0x7f54280b6640 2024-04-18T10:06:08.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f542cc5f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f54280b4f90 con 0x7f54280b6260 2024-04-18T10:06:08.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f5426d9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f54280b6640 0x7f54280b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f542659c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54280b6260 0x7f5428148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f5425d9b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f54281487a0 0x7f54281419a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.62:53720/0 (socket says 172.21.15.62:53720) 2024-04-18T10:06:08.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f5426d9d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f54280b6640 0x7f54280b5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.62:42118/0 (socket says 172.21.15.62:42118) 2024-04-18T10:06:08.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f5425d9b640 1 -- 172.21.15.62:0/3407872348 learned_addr learned my addr 172.21.15.62:0/3407872348 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:08.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f542659c640 1 -- 172.21.15.62:0/3407872348 >> v2:172.21.15.145:3300/0 conn(0x7f54280b6640 msgr2=0x7f54280b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:08.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f542659c640 1 --2- 172.21.15.62:0/3407872348 >> v2:172.21.15.145:3300/0 conn(0x7f54280b6640 0x7f54280b5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.220 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f542659c640 1 -- 172.21.15.62:0/3407872348 >> v1:172.21.15.145:6789/0 conn(0x7f54281487a0 legacy=0x7f54281419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.218+0000 7f542659c640 1 -- 172.21.15.62:0/3407872348 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54280b4c20 con 0x7f54280b6260 2024-04-18T10:06:08.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542659c640 1 --2- 172.21.15.62:0/3407872348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54280b6260 0x7f5428148260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f54180097c0 tx=0x7f541801c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98462f00ee598107 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:08.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542559a640 1 -- 172.21.15.62:0/3407872348 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54180b0070 con 0x7f54280b6260 2024-04-18T10:06:08.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542559a640 1 -- 172.21.15.62:0/3407872348 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54180ab600 con 0x7f54280b6260 2024-04-18T10:06:08.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542559a640 1 -- 172.21.15.62:0/3407872348 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54180ab9d0 con 0x7f54280b6260 2024-04-18T10:06:08.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 -- 172.21.15.62:0/3407872348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54280b6260 msgr2=0x7f5428148260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 --2- 172.21.15.62:0/3407872348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54280b6260 0x7f5428148260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f54180097c0 tx=0x7f541801c380 comp rx=0 tx=0).stop 2024-04-18T10:06:08.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 -- 172.21.15.62:0/3407872348 shutdown_connections 2024-04-18T10:06:08.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 --2- 172.21.15.62:0/3407872348 >> v2:172.21.15.145:3300/0 conn(0x7f54280b6640 0x7f54280b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 --2- 172.21.15.62:0/3407872348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54280b6260 0x7f5428148260 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 -- 172.21.15.62:0/3407872348 >> 172.21.15.62:0/3407872348 conn(0x7f54280bd5c0 msgr2=0x7f54280bd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:08.222 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 -- 172.21.15.62:0/3407872348 shutdown_connections 2024-04-18T10:06:08.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 -- 172.21.15.62:0/3407872348 wait complete. 2024-04-18T10:06:08.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 Processor -- start 2024-04-18T10:06:08.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 -- start start 2024-04-18T10:06:08.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54280b6260 0x7f5428144630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54280b6640 0x7f542814b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54281487a0 0x7f5428143470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5428155240 con 0x7f54280b6260 2024-04-18T10:06:08.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f54281550c0 con 0x7f54281487a0 2024-04-18T10:06:08.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.222+0000 7f542cc5f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f54281553c0 con 0x7f54280b6640 2024-04-18T10:06:08.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f542659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54280b6260 0x7f5428144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f542659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54280b6260 0x7f5428144630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53512/0 (socket says 172.21.15.62:53512) 2024-04-18T10:06:08.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f542659c640 1 -- 172.21.15.62:0/3050284160 learned_addr learned my addr 172.21.15.62:0/3050284160 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:08.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f5426d9d640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54281487a0 0x7f5428143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.225 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f5425d9b640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54280b6640 0x7f542814b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f542659c640 1 -- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54280b6640 msgr2=0x7f542814b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f542659c640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54280b6640 0x7f542814b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f542659c640 1 -- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54281487a0 msgr2=0x7f5428143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f542659c640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54281487a0 0x7f5428143470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f542659c640 1 -- 172.21.15.62:0/3050284160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542813ee80 con 0x7f54280b6260 2024-04-18T10:06:08.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f542659c640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54280b6260 0x7f5428144630 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5418019fb0 tx=0x7f541801abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:08.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f5416ffd640 1 -- 172.21.15.62:0/3050284160 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54180b0050 con 0x7f54280b6260 2024-04-18T10:06:08.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f5416ffd640 1 -- 172.21.15.62:0/3050284160 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54180bc070 con 0x7f54280b6260 2024-04-18T10:06:08.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f542cc5f640 1 -- 172.21.15.62:0/3050284160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f542813f160 con 0x7f54280b6260 2024-04-18T10:06:08.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f5416ffd640 1 -- 172.21.15.62:0/3050284160 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418006e20 con 0x7f54280b6260 2024-04-18T10:06:08.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f542cc5f640 1 -- 172.21.15.62:0/3050284160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54281e1540 con 0x7f54280b6260 2024-04-18T10:06:08.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f542cc5f640 1 -- 172.21.15.62:0/3050284160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5428073bf0 con 0x7f54280b6260 2024-04-18T10:06:08.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f5416ffd640 1 -- 172.21.15.62:0/3050284160 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f54180b5830 con 0x7f54280b6260 2024-04-18T10:06:08.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f5416ffd640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f54080683b0 0x7f540806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f5416ffd640 1 -- 172.21.15.62:0/3050284160 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5418120f80 con 0x7f54280b6260 2024-04-18T10:06:08.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.226+0000 7f5425d9b640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f54080683b0 0x7f540806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.230+0000 7f5425d9b640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f54080683b0 0x7f540806a870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54281e4df0 tx=0x7f541c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:08.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.234+0000 7f5416ffd640 1 -- 172.21.15.62:0/3050284160 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f542813ee80 con 0x7f54280b6260 2024-04-18T10:06:08.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.426+0000 7f542cc5f640 1 -- 172.21.15.62:0/3050284160 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5428068b90 con 0x7f54280b6260 2024-04-18T10:06:08.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.426+0000 7f5416ffd640 1 -- 172.21.15.62:0/3050284160 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3527 (secure 0 0 0) 0x7f5418016090 con 0x7f54280b6260 2024-04-18T10:06:08.427 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:08.427 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/904929673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":904929673},{"type":"v1","addr":"172.21.15.145:6835","nonce":904929673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:05:58.588100+0000","modified":"2024-04-18T10:06:01.605447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2120622858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2120622858},{"type":"v1","addr":"172.21.15.62:6833","nonce":2120622858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-18T10:06:08.427 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-04-18T10:06:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.430+0000 7f542cc5f640 1 -- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f54080683b0 msgr2=0x7f540806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.430+0000 7f542cc5f640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f54080683b0 0x7f540806a870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54281e4df0 tx=0x7f541c009040 comp rx=0 tx=0).stop 2024-04-18T10:06:08.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.430+0000 7f542cc5f640 1 -- 172.21.15.62:0/3050284160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54280b6260 msgr2=0x7f5428144630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.430+0000 7f542cc5f640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54280b6260 0x7f5428144630 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5418019fb0 tx=0x7f541801abc0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.430+0000 7f542cc5f640 1 -- 172.21.15.62:0/3050284160 shutdown_connections 2024-04-18T10:06:08.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.430+0000 7f542cc5f640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f54080683b0 0x7f540806a870 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.430+0000 7f542cc5f640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54281487a0 0x7f5428143470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.430+0000 7f542cc5f640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54280b6640 0x7f542814b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.430+0000 7f542cc5f640 1 --2- 172.21.15.62:0/3050284160 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54280b6260 0x7f5428144630 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.430+0000 7f542cc5f640 1 -- 172.21.15.62:0/3050284160 >> 172.21.15.62:0/3050284160 conn(0x7f54280bd5c0 msgr2=0x7f54280b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:08.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.430+0000 7f542cc5f640 1 -- 172.21.15.62:0/3050284160 shutdown_connections 2024-04-18T10:06:08.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.430+0000 7f542cc5f640 1 -- 172.21.15.62:0/3050284160 wait complete. 2024-04-18T10:06:08.445 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:08.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.566+0000 7f2877243640 1 Processor -- start 2024-04-18T10:06:08.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.566+0000 7f2877243640 1 -- start start 2024-04-18T10:06:08.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.566+0000 7f2877243640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f28700afe00 0x7f287009e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f2877243640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f287009cb10 con 0x7f287009c730 2024-04-18T10:06:08.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f2877243640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28700b02e0 con 0x7f287014b370 2024-04-18T10:06:08.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f2877243640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f28700b9ea0 con 0x7f28700afe00 2024-04-18T10:06:08.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f2874fdf640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f287009c730 0x7f28700986a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55018/0 (socket says 172.21.15.62:55018) 2024-04-18T10:06:08.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f2874fdf640 1 -- 172.21.15.62:0/48329408 learned_addr learned my addr 172.21.15.62:0/48329408 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:08.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286ffff640 1 --2- 172.21.15.62:0/48329408 >> v2:172.21.15.145:3300/0 conn(0x7f28700afe00 0x7f287009e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 486324330 0 0) 0x7f287009cb10 con 0x7f287009c730 2024-04-18T10:06:08.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f285c003530 con 0x7f287009c730 2024-04-18T10:06:08.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 270451512 0 0) 0x7f285c003530 con 0x7f287009c730 2024-04-18T10:06:08.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 >> v2:172.21.15.145:3300/0 conn(0x7f28700afe00 msgr2=0x7f287009e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:08.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 --2- 172.21.15.62:0/48329408 >> v2:172.21.15.145:3300/0 conn(0x7f28700afe00 0x7f287009e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 >> v1:172.21.15.145:6789/0 conn(0x7f287014b370 legacy=0x7f28701526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28700b9b30 con 0x7f287009c730 2024-04-18T10:06:08.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f2860001680 con 0x7f287009c730 2024-04-18T10:06:08.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 >> v1:172.21.15.62:6789/0 conn(0x7f287009c730 legacy=0x7f28700986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 --2- 172.21.15.62:0/48329408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c0039c0 0x7f285c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 --2- 172.21.15.62:0/48329408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f285c004810 0x7f285c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 --2- 172.21.15.62:0/48329408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f285c0051d0 0x7f285c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f28700b9ea0 con 0x7f285c0039c0 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f285c00c070 con 0x7f285c0051d0 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f285c00c270 con 0x7f285c004810 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f2874fdf640 1 --2- 172.21.15.62:0/48329408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c0039c0 0x7f285c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f2874fdf640 1 -- 172.21.15.62:0/48329408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f285c004810 msgr2=0x7f285c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f2874fdf640 1 --2- 172.21.15.62:0/48329408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f285c004810 0x7f285c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f28757e0640 1 --2- 172.21.15.62:0/48329408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f285c0051d0 0x7f285c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f2874fdf640 1 -- 172.21.15.62:0/48329408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f285c0051d0 msgr2=0x7f285c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f2874fdf640 1 --2- 172.21.15.62:0/48329408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f285c0051d0 0x7f285c00b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f2874fdf640 1 -- 172.21.15.62:0/48329408 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285c003530 con 0x7f285c0039c0 2024-04-18T10:06:08.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f28757e0640 1 -- 172.21.15.62:0/48329408 reap_dead start 2024-04-18T10:06:08.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f2874fdf640 1 --2- 172.21.15.62:0/48329408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c0039c0 0x7f285c003de0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f286000c1c0 tx=0x7f286001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f5b48f7e6c4b3e server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:08.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2860007d00 con 0x7f285c0039c0 2024-04-18T10:06:08.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28600a9020 con 0x7f285c0039c0 2024-04-18T10:06:08.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.570+0000 7f286f7fe640 1 -- 172.21.15.62:0/48329408 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2860005b00 con 0x7f285c0039c0 2024-04-18T10:06:08.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 -- 172.21.15.62:0/48329408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c0039c0 msgr2=0x7f285c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 --2- 172.21.15.62:0/48329408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c0039c0 0x7f285c003de0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f286000c1c0 tx=0x7f286001cc60 comp rx=0 tx=0).stop 2024-04-18T10:06:08.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 -- 172.21.15.62:0/48329408 shutdown_connections 2024-04-18T10:06:08.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 --2- 172.21.15.62:0/48329408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f285c0039c0 0x7f285c003de0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 -- 172.21.15.62:0/48329408 >> 172.21.15.62:0/48329408 conn(0x7f287006c0d0 msgr2=0x7f28700b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:08.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 -- 172.21.15.62:0/48329408 shutdown_connections 2024-04-18T10:06:08.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 -- 172.21.15.62:0/48329408 wait complete. 2024-04-18T10:06:08.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 Processor -- start 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 -- start start 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f285c0039c0 0x7f2870072b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2870071b80 0x7f287009b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28701e3ea0 0x7f28701e4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f287006d620 con 0x7f2870071b80 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f287006d4a0 con 0x7f285c0039c0 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f287006d7a0 con 0x7f28701e3ea0 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f286ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2870071b80 0x7f287009b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f286ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2870071b80 0x7f287009b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53540/0 (socket says 172.21.15.62:53540) 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f286ffff640 1 -- 172.21.15.62:0/842081120 learned_addr learned my addr 172.21.15.62:0/842081120 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2874fdf640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f285c0039c0 0x7f2870072b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f28757e0640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28701e3ea0 0x7f28701e4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f286ffff640 1 -- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28701e3ea0 msgr2=0x7f28701e4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f286ffff640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28701e3ea0 0x7f28701e4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f286ffff640 1 -- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f285c0039c0 msgr2=0x7f2870072b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f286ffff640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f285c0039c0 0x7f2870072b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f286ffff640 1 -- 172.21.15.62:0/842081120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28700720b0 con 0x7f2870071b80 2024-04-18T10:06:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2874fdf640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f285c0039c0 0x7f2870072b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f286ffff640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2870071b80 0x7f287009b0b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2858009db0 tx=0x7f285800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f286d7fa640 1 -- 172.21.15.62:0/842081120 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858018020 con 0x7f2870071b80 2024-04-18T10:06:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f286d7fa640 1 -- 172.21.15.62:0/842081120 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2858014480 con 0x7f2870071b80 2024-04-18T10:06:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 -- 172.21.15.62:0/842081120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2870073050 con 0x7f2870071b80 2024-04-18T10:06:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f286d7fa640 1 -- 172.21.15.62:0/842081120 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858014870 con 0x7f2870071b80 2024-04-18T10:06:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 -- 172.21.15.62:0/842081120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2870070900 con 0x7f2870071b80 2024-04-18T10:06:08.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.574+0000 7f2877243640 1 -- 172.21.15.62:0/842081120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2870068b90 con 0x7f2870071b80 2024-04-18T10:06:08.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.578+0000 7f286d7fa640 1 -- 172.21.15.62:0/842081120 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f285801b020 con 0x7f2870071b80 2024-04-18T10:06:08.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.578+0000 7f286d7fa640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f28500683b0 0x7f285006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.578+0000 7f286d7fa640 1 -- 172.21.15.62:0/842081120 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2858087300 con 0x7f2870071b80 2024-04-18T10:06:08.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.578+0000 7f2874fdf640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f28500683b0 0x7f285006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.578+0000 7f2874fdf640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f28500683b0 0x7f285006a870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f28600aa000 tx=0x7f28700b5eb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:08.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.582+0000 7f286d7fa640 1 -- 172.21.15.62:0/842081120 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2858022070 con 0x7f2870071b80 2024-04-18T10:06:08.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.742+0000 7f2877243640 1 -- 172.21.15.62:0/842081120 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f287014ae20 con 0x7f2870071b80 2024-04-18T10:06:08.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f286d7fa640 1 -- 172.21.15.62:0/842081120 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f28700720b0 con 0x7f2870071b80 2024-04-18T10:06:08.744 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:08.744 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:08.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f2877243640 1 -- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f28500683b0 msgr2=0x7f285006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f2877243640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f28500683b0 0x7f285006a870 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f28600aa000 tx=0x7f28700b5eb0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f2877243640 1 -- 172.21.15.62:0/842081120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2870071b80 msgr2=0x7f287009b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f2877243640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2870071b80 0x7f287009b0b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2858009db0 tx=0x7f285800c490 comp rx=0 tx=0).stop 2024-04-18T10:06:08.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f2877243640 1 -- 172.21.15.62:0/842081120 shutdown_connections 2024-04-18T10:06:08.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f2877243640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f28500683b0 0x7f285006a870 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f2877243640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28701e3ea0 0x7f28701e4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f2877243640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2870071b80 0x7f287009b0b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f2877243640 1 --2- 172.21.15.62:0/842081120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f285c0039c0 0x7f2870072b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f2877243640 1 -- 172.21.15.62:0/842081120 >> 172.21.15.62:0/842081120 conn(0x7f287006c0d0 msgr2=0x7f2870149a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:08.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f2877243640 1 -- 172.21.15.62:0/842081120 shutdown_connections 2024-04-18T10:06:08.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.746+0000 7f2877243640 1 -- 172.21.15.62:0/842081120 wait complete. 2024-04-18T10:06:08.763 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-18T10:05:56.782649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-18T10:05:57.541623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:08.763 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:08.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1ef102640 1 Processor -- start 2024-04-18T10:06:08.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1ef102640 1 -- start start 2024-04-18T10:06:08.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1ef102640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe1e8146690 0x7fe1e809e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1ef102640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe1e806e490 0x7fe1e809e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1ef102640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1e806d4f0 con 0x7fe1e80b4c90 2024-04-18T10:06:08.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1ef102640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fe1e806d2a0 con 0x7fe1e8146690 2024-04-18T10:06:08.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1ef102640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fe1e806d7d0 con 0x7fe1e806e490 2024-04-18T10:06:08.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1e7fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe1e8146690 0x7fe1e809e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1ece9e640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe1e80b4c90 0x7fe1e8145f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55022/0 (socket says 172.21.15.62:55022) 2024-04-18T10:06:08.889 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1ece9e640 1 -- 172.21.15.62:0/535102597 learned_addr learned my addr 172.21.15.62:0/535102597 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:08.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1ed69f640 1 --2- 172.21.15.62:0/535102597 >> v2:172.21.15.145:3300/0 conn(0x7fe1e806e490 0x7fe1e809e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1e7fff640 1 -- 172.21.15.62:0/535102597 >> v2:172.21.15.145:3300/0 conn(0x7fe1e806e490 msgr2=0x7fe1e809e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:08.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1e7fff640 1 --2- 172.21.15.62:0/535102597 >> v2:172.21.15.145:3300/0 conn(0x7fe1e806e490 0x7fe1e809e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1e7fff640 1 -- 172.21.15.62:0/535102597 >> v1:172.21.15.62:6789/0 conn(0x7fe1e80b4c90 legacy=0x7fe1e8145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1e77fe640 1 -- 172.21.15.62:0/535102597 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1717580065 0 0) 0x7fe1e806d4f0 con 0x7fe1e80b4c90 2024-04-18T10:06:08.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1e7fff640 1 -- 172.21.15.62:0/535102597 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e809c570 con 0x7fe1e8146690 2024-04-18T10:06:08.890 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.890+0000 7fe1ed69f640 1 --2- 172.21.15.62:0/535102597 >> v2:172.21.15.145:3300/0 conn(0x7fe1e806e490 0x7fe1e809e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:08.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1e7fff640 1 --2- 172.21.15.62:0/535102597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8146690 0x7fe1e809e090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe1e80b54d0 tx=0x7fe1d401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647cf43efd1c8688 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:08.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1e77fe640 1 -- 172.21.15.62:0/535102597 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40b0070 con 0x7fe1e8146690 2024-04-18T10:06:08.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1e77fe640 1 -- 172.21.15.62:0/535102597 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1d40ab600 con 0x7fe1e8146690 2024-04-18T10:06:08.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1e77fe640 1 -- 172.21.15.62:0/535102597 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40ab9d0 con 0x7fe1e8146690 2024-04-18T10:06:08.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 -- 172.21.15.62:0/535102597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8146690 msgr2=0x7fe1e809e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 --2- 172.21.15.62:0/535102597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8146690 0x7fe1e809e090 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe1e80b54d0 tx=0x7fe1d401c3f0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 -- 172.21.15.62:0/535102597 shutdown_connections 2024-04-18T10:06:08.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 --2- 172.21.15.62:0/535102597 >> v2:172.21.15.145:3300/0 conn(0x7fe1e806e490 0x7fe1e809e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 --2- 172.21.15.62:0/535102597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e8146690 0x7fe1e809e090 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 -- 172.21.15.62:0/535102597 >> 172.21.15.62:0/535102597 conn(0x7fe1e80b9690 msgr2=0x7fe1e80b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:08.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 -- 172.21.15.62:0/535102597 shutdown_connections 2024-04-18T10:06:08.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 -- 172.21.15.62:0/535102597 wait complete. 2024-04-18T10:06:08.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 Processor -- start 2024-04-18T10:06:08.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 -- start start 2024-04-18T10:06:08.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1e806e490 0x7fe1e81e36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.896 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e80b4c90 0x7fe1e80afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1e8146690 0x7fe1e8138210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e8152fc0 con 0x7fe1e80b4c90 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e8152e40 con 0x7fe1e8146690 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1ef102640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fe1e8153140 con 0x7fe1e806e490 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1e7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e80b4c90 0x7fe1e80afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1e7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e80b4c90 0x7fe1e80afeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53566/0 (socket says 172.21.15.62:53566) 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.894+0000 7fe1e7fff640 1 -- 172.21.15.62:0/1363887293 learned_addr learned my addr 172.21.15.62:0/1363887293 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1ed69f640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1e8146690 0x7fe1e8138210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1ece9e640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1e806e490 0x7fe1e81e36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1e7fff640 1 -- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1e806e490 msgr2=0x7fe1e81e36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1e7fff640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1e806e490 0x7fe1e81e36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1e7fff640 1 -- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1e8146690 msgr2=0x7fe1e8138210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:08.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1e7fff640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1e8146690 0x7fe1e8138210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1e7fff640 1 -- 172.21.15.62:0/1363887293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e80af740 con 0x7fe1e80b4c90 2024-04-18T10:06:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1ed69f640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1e8146690 0x7fe1e8138210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1e7fff640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e80b4c90 0x7fe1e80afeb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40ae040 tx=0x7fe1d4019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1e57fa640 1 -- 172.21.15.62:0/1363887293 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40b8020 con 0x7fe1e80b4c90 2024-04-18T10:06:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1e57fa640 1 -- 172.21.15.62:0/1363887293 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1d40b4610 con 0x7fe1e80b4c90 2024-04-18T10:06:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1ef102640 1 -- 172.21.15.62:0/1363887293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e81499a0 con 0x7fe1e80b4c90 2024-04-18T10:06:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1e57fa640 1 -- 172.21.15.62:0/1363887293 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1d40b0040 con 0x7fe1e80b4c90 2024-04-18T10:06:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1ef102640 1 -- 172.21.15.62:0/1363887293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e813f4b0 con 0x7fe1e80b4c90 2024-04-18T10:06:08.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1ef102640 1 -- 172.21.15.62:0/1363887293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1e8149ce0 con 0x7fe1e80b4c90 2024-04-18T10:06:08.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1e57fa640 1 -- 172.21.15.62:0/1363887293 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe1d40bb020 con 0x7fe1e80b4c90 2024-04-18T10:06:08.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1e57fa640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe1cc068380 0x7fe1cc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:08.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1e57fa640 1 -- 172.21.15.62:0/1363887293 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe1d41205f0 con 0x7fe1e80b4c90 2024-04-18T10:06:08.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1ece9e640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe1cc068380 0x7fe1cc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:08.899 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.898+0000 7fe1ece9e640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe1cc068380 0x7fe1cc06a840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc000f70 tx=0x7fe1dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:08.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:08.906+0000 7fe1e57fa640 1 -- 172.21.15.62:0/1363887293 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe1d40c2030 con 0x7fe1e80b4c90 2024-04-18T10:06:09.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.066+0000 7fe1ef102640 1 -- 172.21.15.62:0/1363887293 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe1e8068c00 con 0x7fe1e80b4c90 2024-04-18T10:06:09.067 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1e57fa640 1 -- 172.21.15.62:0/1363887293 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fe1d4016070 con 0x7fe1e80b4c90 2024-04-18T10:06:09.067 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:09.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1ef102640 1 -- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe1cc068380 msgr2=0x7fe1cc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1ef102640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe1cc068380 0x7fe1cc06a840 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc000f70 tx=0x7fe1dc009040 comp rx=0 tx=0).stop 2024-04-18T10:06:09.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1ef102640 1 -- 172.21.15.62:0/1363887293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e80b4c90 msgr2=0x7fe1e80afeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1ef102640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e80b4c90 0x7fe1e80afeb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe1d40ae040 tx=0x7fe1d4019190 comp rx=0 tx=0).stop 2024-04-18T10:06:09.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1ef102640 1 -- 172.21.15.62:0/1363887293 shutdown_connections 2024-04-18T10:06:09.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1ef102640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe1cc068380 0x7fe1cc06a840 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1ef102640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1e8146690 0x7fe1e8138210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1ef102640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1e80b4c90 0x7fe1e80afeb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1ef102640 1 --2- 172.21.15.62:0/1363887293 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1e806e490 0x7fe1e81e36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1ef102640 1 -- 172.21.15.62:0/1363887293 >> 172.21.15.62:0/1363887293 conn(0x7fe1e80b9690 msgr2=0x7fe1e80b6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:09.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1ef102640 1 -- 172.21.15.62:0/1363887293 shutdown_connections 2024-04-18T10:06:09.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.070+0000 7fe1ef102640 1 -- 172.21.15.62:0/1363887293 wait complete. 2024-04-18T10:06:09.087 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-18T10:06:09.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb45e876640 1 Processor -- start 2024-04-18T10:06:09.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb45e876640 1 -- start start 2024-04-18T10:06:09.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb45e876640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb45809ae40 0x7fb4580b7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb45e876640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4580b75b0 0x7fb4581491f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb45e876640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb45809b220 con 0x7fb458149730 2024-04-18T10:06:09.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb45e876640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fb45806f300 con 0x7fb45809ae40 2024-04-18T10:06:09.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb45e876640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fb45806e820 con 0x7fb4580b75b0 2024-04-18T10:06:09.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb457fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb45809ae40 0x7fb4580b7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb457fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb45809ae40 0x7fb4580b7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53568/0 (socket says 172.21.15.62:53568) 2024-04-18T10:06:09.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb457fff640 1 -- 172.21.15.62:0/3681094701 learned_addr learned my addr 172.21.15.62:0/3681094701 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:09.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb4577fe640 1 --2- 172.21.15.62:0/3681094701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4580b75b0 0x7fb4581491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb457fff640 1 -- 172.21.15.62:0/3681094701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4580b75b0 msgr2=0x7fb4581491f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:09.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb457fff640 1 --2- 172.21.15.62:0/3681094701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4580b75b0 0x7fb4581491f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb457fff640 1 -- 172.21.15.62:0/3681094701 >> v1:172.21.15.145:6789/0 conn(0x7fb458149730 legacy=0x7fb458073580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:09.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.210+0000 7fb457fff640 1 -- 172.21.15.62:0/3681094701 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45806eb00 con 0x7fb45809ae40 2024-04-18T10:06:09.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb457fff640 1 --2- 172.21.15.62:0/3681094701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb45809ae40 0x7fb4580b7070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb440009870 tx=0x7fb44001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1961c995606c2167 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:09.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb456ffd640 1 -- 172.21.15.62:0/3681094701 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4400aa620 con 0x7fb45809ae40 2024-04-18T10:06:09.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb456ffd640 1 -- 172.21.15.62:0/3681094701 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4400af070 con 0x7fb45809ae40 2024-04-18T10:06:09.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb456ffd640 1 -- 172.21.15.62:0/3681094701 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4400aa9e0 con 0x7fb45809ae40 2024-04-18T10:06:09.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 -- 172.21.15.62:0/3681094701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb45809ae40 msgr2=0x7fb4580b7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 --2- 172.21.15.62:0/3681094701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb45809ae40 0x7fb4580b7070 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb440009870 tx=0x7fb44001c4a0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.212 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 -- 172.21.15.62:0/3681094701 shutdown_connections 2024-04-18T10:06:09.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 --2- 172.21.15.62:0/3681094701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4580b75b0 0x7fb4581491f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 --2- 172.21.15.62:0/3681094701 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb45809ae40 0x7fb4580b7070 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 -- 172.21.15.62:0/3681094701 >> 172.21.15.62:0/3681094701 conn(0x7fb4580ba780 msgr2=0x7fb4580bab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:09.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 -- 172.21.15.62:0/3681094701 shutdown_connections 2024-04-18T10:06:09.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 -- 172.21.15.62:0/3681094701 wait complete. 2024-04-18T10:06:09.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 Processor -- start 2024-04-18T10:06:09.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 -- start start 2024-04-18T10:06:09.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb45809ae40 0x7fb4581e1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4580b75b0 0x7fb45813c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb458149730 0x7fb45813ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb4581530d0 con 0x7fb45809ae40 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fb458152f50 con 0x7fb4580b75b0 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fb458153250 con 0x7fb458149730 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb457fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb45809ae40 0x7fb4581e1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb457fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb45809ae40 0x7fb4581e1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53578/0 (socket says 172.21.15.62:53578) 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb457fff640 1 -- 172.21.15.62:0/2246058671 learned_addr learned my addr 172.21.15.62:0/2246058671 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb4577fe640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4580b75b0 0x7fb45813c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45ce13640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb458149730 0x7fb45813ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb457fff640 1 -- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb458149730 msgr2=0x7fb45813ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb457fff640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb458149730 0x7fb45813ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb457fff640 1 -- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4580b75b0 msgr2=0x7fb45813c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb457fff640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4580b75b0 0x7fb45813c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb457fff640 1 -- 172.21.15.62:0/2246058671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4580b4650 con 0x7fb45809ae40 2024-04-18T10:06:09.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb4577fe640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4580b75b0 0x7fb45813c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:09.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb457fff640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb45809ae40 0x7fb4581e1e20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb4400ac040 tx=0x7fb44001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:09.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb454ff9640 1 -- 172.21.15.62:0/2246058671 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4400aade0 con 0x7fb45809ae40 2024-04-18T10:06:09.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb454ff9640 1 -- 172.21.15.62:0/2246058671 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4400bd3f0 con 0x7fb45809ae40 2024-04-18T10:06:09.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 -- 172.21.15.62:0/2246058671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb458143e00 con 0x7fb45809ae40 2024-04-18T10:06:09.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb454ff9640 1 -- 172.21.15.62:0/2246058671 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4400bd7e0 con 0x7fb45809ae40 2024-04-18T10:06:09.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.214+0000 7fb45e876640 1 -- 172.21.15.62:0/2246058671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4580b26e0 con 0x7fb45809ae40 2024-04-18T10:06:09.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.218+0000 7fb45e876640 1 -- 172.21.15.62:0/2246058671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb458069230 con 0x7fb45809ae40 2024-04-18T10:06:09.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.218+0000 7fb454ff9640 1 -- 172.21.15.62:0/2246058671 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb4400c6440 con 0x7fb45809ae40 2024-04-18T10:06:09.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.218+0000 7fb454ff9640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb438068380 0x7fb43806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.218+0000 7fb454ff9640 1 -- 172.21.15.62:0/2246058671 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb440120e00 con 0x7fb45809ae40 2024-04-18T10:06:09.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.218+0000 7fb4577fe640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb438068380 0x7fb43806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.218+0000 7fb4577fe640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb438068380 0x7fb43806a840 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb458072e90 tx=0x7fb448008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:09.221 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.222+0000 7fb454ff9640 1 -- 172.21.15.62:0/2246058671 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb440016150 con 0x7fb45809ae40 2024-04-18T10:06:09.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 -- 172.21.15.62:0/2246058671 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb45813d200 con 0x7fb45809ae40 2024-04-18T10:06:09.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb454ff9640 1 -- 172.21.15.62:0/2246058671 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fb4400f1450 con 0x7fb45809ae40 2024-04-18T10:06:09.380 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-18T10:06:09.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 -- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb438068380 msgr2=0x7fb43806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb438068380 0x7fb43806a840 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb458072e90 tx=0x7fb448008040 comp rx=0 tx=0).stop 2024-04-18T10:06:09.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 -- 172.21.15.62:0/2246058671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb45809ae40 msgr2=0x7fb4581e1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb45809ae40 0x7fb4581e1e20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fb4400ac040 tx=0x7fb44001ace0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 -- 172.21.15.62:0/2246058671 shutdown_connections 2024-04-18T10:06:09.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb438068380 0x7fb43806a840 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb458149730 0x7fb45813ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4580b75b0 0x7fb45813c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 --2- 172.21.15.62:0/2246058671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb45809ae40 0x7fb4581e1e20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 -- 172.21.15.62:0/2246058671 >> 172.21.15.62:0/2246058671 conn(0x7fb4580ba780 msgr2=0x7fb4580b5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:09.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 -- 172.21.15.62:0/2246058671 shutdown_connections 2024-04-18T10:06:09.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.382+0000 7fb45e876640 1 -- 172.21.15.62:0/2246058671 wait complete. 2024-04-18T10:06:09.398 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-18T10:06:09.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f567bf5f640 1 Processor -- start 2024-04-18T10:06:09.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f567bf5f640 1 -- start start 2024-04-18T10:06:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f567bf5f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f56740b01a0 0x7f567406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f567bf5f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56740ae4e0 0x7f567413e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f567bf5f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56740b0580 con 0x7f56740737c0 2024-04-18T10:06:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f567bf5f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f56740b59d0 con 0x7f56740b01a0 2024-04-18T10:06:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f567bf5f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f56740ae970 con 0x7f56740ae4e0 2024-04-18T10:06:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f567a4fc640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f56740737c0 0x7f5674148100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55024/0 (socket says 172.21.15.62:55024) 2024-04-18T10:06:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f567a4fc640 1 -- 172.21.15.62:0/1679511857 learned_addr learned my addr 172.21.15.62:0/1679511857 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5679cfb640 1 --2- 172.21.15.62:0/1679511857 >> v2:172.21.15.145:3300/0 conn(0x7f56740b01a0 0x7f567406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f56794fa640 1 --2- 172.21.15.62:0/1679511857 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56740ae4e0 0x7f567413e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4280021353 0 0) 0x7f56740b0580 con 0x7f56740737c0 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5660003530 con 0x7f56740737c0 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1084887116 0 0) 0x7f5660003530 con 0x7f56740737c0 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56740ae4e0 msgr2=0x7f567413e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 --2- 172.21.15.62:0/1679511857 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56740ae4e0 0x7f567413e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 >> v2:172.21.15.145:3300/0 conn(0x7f56740b01a0 msgr2=0x7f567406d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 --2- 172.21.15.62:0/1679511857 >> v2:172.21.15.145:3300/0 conn(0x7f56740b01a0 0x7f567406d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56740b53a0 con 0x7f56740737c0 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f566c001a00 con 0x7f56740737c0 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 >> v1:172.21.15.62:6789/0 conn(0x7f56740737c0 legacy=0x7f5674148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 --2- 172.21.15.62:0/1679511857 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56600039c0 0x7f5660003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 --2- 172.21.15.62:0/1679511857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5660004810 0x7f5660004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 --2- 172.21.15.62:0/1679511857 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56600051d0 0x7f566000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f56740b59d0 con 0x7f5660004810 2024-04-18T10:06:09.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f56740ae970 con 0x7f56600039c0 2024-04-18T10:06:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f566000bf10 con 0x7f56600051d0 2024-04-18T10:06:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f56794fa640 1 --2- 172.21.15.62:0/1679511857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5660004810 0x7f5660004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f56794fa640 1 -- 172.21.15.62:0/1679511857 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56600051d0 msgr2=0x7f566000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f56794fa640 1 --2- 172.21.15.62:0/1679511857 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56600051d0 0x7f566000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f56794fa640 1 -- 172.21.15.62:0/1679511857 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56600039c0 msgr2=0x7f5660003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f56794fa640 1 --2- 172.21.15.62:0/1679511857 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56600039c0 0x7f5660003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.522+0000 7f56794fa640 1 -- 172.21.15.62:0/1679511857 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5660003530 con 0x7f5660004810 2024-04-18T10:06:09.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f56794fa640 1 --2- 172.21.15.62:0/1679511857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5660004810 0x7f5660004c90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5664009b10 tx=0x7f566401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d7ebea4e248ff0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:09.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56640096b0 con 0x7f5660004810 2024-04-18T10:06:09.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56640a64c0 con 0x7f5660004810 2024-04-18T10:06:09.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f5678cf9640 1 -- 172.21.15.62:0/1679511857 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56640a68b0 con 0x7f5660004810 2024-04-18T10:06:09.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 -- 172.21.15.62:0/1679511857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5660004810 msgr2=0x7f5660004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 --2- 172.21.15.62:0/1679511857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5660004810 0x7f5660004c90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5664009b10 tx=0x7f566401cab0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567a4fc640 1 -- 172.21.15.62:0/1679511857 reap_dead start 2024-04-18T10:06:09.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 -- 172.21.15.62:0/1679511857 shutdown_connections 2024-04-18T10:06:09.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 -- 172.21.15.62:0/1679511857 >> 172.21.15.62:0/1679511857 conn(0x7f56740b9690 msgr2=0x7f56740b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:09.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 -- 172.21.15.62:0/1679511857 shutdown_connections 2024-04-18T10:06:09.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 -- 172.21.15.62:0/1679511857 wait complete. 2024-04-18T10:06:09.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 Processor -- start 2024-04-18T10:06:09.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 -- start start 2024-04-18T10:06:09.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5660004810 0x7f56741e3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56741e39b0 0x7f567409db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567409e0b0 0x7f5674072260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f567413e980 con 0x7f567409e0b0 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f567406e470 con 0x7f5660004810 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f567413eb00 con 0x7f56741e39b0 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567a4fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567409e0b0 0x7f5674072260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567a4fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567409e0b0 0x7f5674072260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53592/0 (socket says 172.21.15.62:53592) 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567a4fc640 1 -- 172.21.15.62:0/1228462892 learned_addr learned my addr 172.21.15.62:0/1228462892 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f5679cfb640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5660004810 0x7f56741e3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567a4fc640 1 -- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56741e39b0 msgr2=0x7f567409db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f56794fa640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56741e39b0 0x7f567409db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567a4fc640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56741e39b0 0x7f567409db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567a4fc640 1 -- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5660004810 msgr2=0x7f56741e3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567a4fc640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5660004810 0x7f56741e3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567a4fc640 1 -- 172.21.15.62:0/1228462892 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56741e3ec0 con 0x7f567409e0b0 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567a4fc640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567409e0b0 0x7f5674072260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f567406ed30 tx=0x7f566c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:09.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f566a7fc640 1 -- 172.21.15.62:0/1228462892 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c00ce50 con 0x7f567409e0b0 2024-04-18T10:06:09.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f566a7fc640 1 -- 172.21.15.62:0/1228462892 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f566c00a5b0 con 0x7f567409e0b0 2024-04-18T10:06:09.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 -- 172.21.15.62:0/1228462892 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5674071d40 con 0x7f567409e0b0 2024-04-18T10:06:09.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f566a7fc640 1 -- 172.21.15.62:0/1228462892 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c00a9a0 con 0x7f567409e0b0 2024-04-18T10:06:09.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.526+0000 7f567bf5f640 1 -- 172.21.15.62:0/1228462892 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f567414ae00 con 0x7f567409e0b0 2024-04-18T10:06:09.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.530+0000 7f567bf5f640 1 -- 172.21.15.62:0/1228462892 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5674068c00 con 0x7f567409e0b0 2024-04-18T10:06:09.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.530+0000 7f566a7fc640 1 -- 172.21.15.62:0/1228462892 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f566c00ab40 con 0x7f567409e0b0 2024-04-18T10:06:09.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.530+0000 7f566a7fc640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56580683b0 0x7f565806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.530+0000 7f566a7fc640 1 -- 172.21.15.62:0/1228462892 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f566c086590 con 0x7f567409e0b0 2024-04-18T10:06:09.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.530+0000 7f5679cfb640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56580683b0 0x7f565806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.530+0000 7f5679cfb640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56580683b0 0x7f565806a870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f56700047c0 tx=0x7f5670008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:09.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.534+0000 7f566a7fc640 1 -- 172.21.15.62:0/1228462892 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f566c056be0 con 0x7f567409e0b0 2024-04-18T10:06:09.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.694+0000 7f567bf5f640 1 -- 172.21.15.62:0/1228462892 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5674071f10 con 0x7f567409e0b0 2024-04-18T10:06:09.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.694+0000 7f566a7fc640 1 -- 172.21.15.62:0/1228462892 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f566c0564e0 con 0x7f567409e0b0 2024-04-18T10:06:09.694 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-04-18T10:06:09.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.698+0000 7f567bf5f640 1 -- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56580683b0 msgr2=0x7f565806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.698+0000 7f567bf5f640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56580683b0 0x7f565806a870 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f56700047c0 tx=0x7f5670008040 comp rx=0 tx=0).stop 2024-04-18T10:06:09.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.698+0000 7f567bf5f640 1 -- 172.21.15.62:0/1228462892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567409e0b0 msgr2=0x7f5674072260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.698+0000 7f567bf5f640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567409e0b0 0x7f5674072260 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f567406ed30 tx=0x7f566c00cc00 comp rx=0 tx=0).stop 2024-04-18T10:06:09.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.698+0000 7f567bf5f640 1 -- 172.21.15.62:0/1228462892 shutdown_connections 2024-04-18T10:06:09.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.698+0000 7f567bf5f640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56580683b0 0x7f565806a870 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.698+0000 7f567bf5f640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f567409e0b0 0x7f5674072260 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.698+0000 7f567bf5f640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56741e39b0 0x7f567409db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.698+0000 7f567bf5f640 1 --2- 172.21.15.62:0/1228462892 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5660004810 0x7f56741e3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.698+0000 7f567bf5f640 1 -- 172.21.15.62:0/1228462892 >> 172.21.15.62:0/1228462892 conn(0x7f56740b9690 msgr2=0x7f5674145350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:09.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.698+0000 7f567bf5f640 1 -- 172.21.15.62:0/1228462892 shutdown_connections 2024-04-18T10:06:09.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.698+0000 7f567bf5f640 1 -- 172.21.15.62:0/1228462892 wait complete. 2024-04-18T10:06:09.713 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:09.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e3a748640 1 Processor -- start 2024-04-18T10:06:09.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e3a748640 1 -- start start 2024-04-18T10:06:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e3a748640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8e34145f20 0x7f8e341514f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e3a748640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e3406e030 con 0x7f8e3409be30 2024-04-18T10:06:09.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e3a748640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e340ba170 con 0x7f8e340b9d90 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e3a748640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f8e3409cd30 con 0x7f8e34145f20 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e337fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8e3409be30 0x7f8e3406d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55036/0 (socket says 172.21.15.62:55036) 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e337fe640 1 -- 172.21.15.62:0/3127894666 learned_addr learned my addr 172.21.15.62:0/3127894666 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e38ce5640 1 --2- 172.21.15.62:0/3127894666 >> v2:172.21.15.145:3300/0 conn(0x7f8e34145f20 0x7f8e341514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1440169006 0 0) 0x7f8e3406e030 con 0x7f8e3409be30 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e1c003530 con 0x7f8e3409be30 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1713617974 0 0) 0x7f8e1c003530 con 0x7f8e3409be30 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 >> v2:172.21.15.145:3300/0 conn(0x7f8e34145f20 msgr2=0x7f8e341514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 --2- 172.21.15.62:0/3127894666 >> v2:172.21.15.145:3300/0 conn(0x7f8e34145f20 0x7f8e341514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 >> v1:172.21.15.145:6789/0 conn(0x7f8e340b9d90 legacy=0x7f8e3406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e3414a060 con 0x7f8e3409be30 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f8e28001680 con 0x7f8e3409be30 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 >> v1:172.21.15.62:6789/0 conn(0x7f8e3409be30 legacy=0x7f8e3406d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 --2- 172.21.15.62:0/3127894666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e1c0039c0 0x7f8e1c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 --2- 172.21.15.62:0/3127894666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e1c004810 0x7f8e1c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 --2- 172.21.15.62:0/3127894666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e1c0052f0 0x7f8e1c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3409cd30 con 0x7f8e1c004810 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c00c1c0 con 0x7f8e1c0052f0 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8e1c00c3c0 con 0x7f8e1c0039c0 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e337fe640 1 --2- 172.21.15.62:0/3127894666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e1c004810 0x7f8e1c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e38ce5640 1 --2- 172.21.15.62:0/3127894666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e1c0052f0 0x7f8e1c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e33fff640 1 --2- 172.21.15.62:0/3127894666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e1c0039c0 0x7f8e1c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e337fe640 1 -- 172.21.15.62:0/3127894666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e1c0039c0 msgr2=0x7f8e1c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e337fe640 1 --2- 172.21.15.62:0/3127894666 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e1c0039c0 0x7f8e1c003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e337fe640 1 -- 172.21.15.62:0/3127894666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e1c0052f0 msgr2=0x7f8e1c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e337fe640 1 --2- 172.21.15.62:0/3127894666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e1c0052f0 0x7f8e1c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.834+0000 7f8e337fe640 1 -- 172.21.15.62:0/3127894666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1c003530 con 0x7f8e1c004810 2024-04-18T10:06:09.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e38ce5640 1 --2- 172.21.15.62:0/3127894666 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e1c0052f0 0x7f8e1c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:09.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e38ce5640 1 -- 172.21.15.62:0/3127894666 reap_dead start 2024-04-18T10:06:09.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e337fe640 1 --2- 172.21.15.62:0/3127894666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e1c004810 0x7f8e1c004c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8e28001060 tx=0x7f8e2801cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eae129931d7b70f server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:09.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e28007d20 con 0x7f8e1c004810 2024-04-18T10:06:09.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e280a9020 con 0x7f8e1c004810 2024-04-18T10:06:09.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e32ffd640 1 -- 172.21.15.62:0/3127894666 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e28005ae0 con 0x7f8e1c004810 2024-04-18T10:06:09.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e3a748640 1 -- 172.21.15.62:0/3127894666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e1c004810 msgr2=0x7f8e1c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e3a748640 1 --2- 172.21.15.62:0/3127894666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e1c004810 0x7f8e1c004c90 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8e28001060 tx=0x7f8e2801cbe0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e3a748640 1 -- 172.21.15.62:0/3127894666 shutdown_connections 2024-04-18T10:06:09.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e3a748640 1 --2- 172.21.15.62:0/3127894666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e1c004810 0x7f8e1c004c90 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e3a748640 1 -- 172.21.15.62:0/3127894666 >> 172.21.15.62:0/3127894666 conn(0x7f8e3406c0d0 msgr2=0x7f8e340b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:09.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e3a748640 1 -- 172.21.15.62:0/3127894666 shutdown_connections 2024-04-18T10:06:09.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.838+0000 7f8e3a748640 1 -- 172.21.15.62:0/3127894666 wait complete. 2024-04-18T10:06:09.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e3a748640 1 Processor -- start 2024-04-18T10:06:09.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e3a748640 1 -- start start 2024-04-18T10:06:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e3a748640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e1c004810 0x7f8e341d1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e3a748640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e341e1380 0x7f8e341e1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e3a748640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e340b2e70 0x7f8e340b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e3a748640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8e340b3e20 con 0x7f8e340b2e70 2024-04-18T10:06:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e3a748640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f8e34146330 con 0x7f8e341e1380 2024-04-18T10:06:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e3a748640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8e340b3fa0 con 0x7f8e1c004810 2024-04-18T10:06:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e38ce5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e340b2e70 0x7f8e340b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e38ce5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e340b2e70 0x7f8e340b32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53614/0 (socket says 172.21.15.62:53614) 2024-04-18T10:06:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e38ce5640 1 -- 172.21.15.62:0/3400319809 learned_addr learned my addr 172.21.15.62:0/3400319809 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e337fe640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e341e1380 0x7f8e341e1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e33fff640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e1c004810 0x7f8e341d1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e38ce5640 1 -- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e1c004810 msgr2=0x7f8e341d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e38ce5640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e1c004810 0x7f8e341d1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e38ce5640 1 -- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e341e1380 msgr2=0x7f8e341e1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e38ce5640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e341e1380 0x7f8e341e1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e38ce5640 1 -- 172.21.15.62:0/3400319809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e341d3ad0 con 0x7f8e340b2e70 2024-04-18T10:06:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e38ce5640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e340b2e70 0x7f8e340b32f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c009f50 tx=0x7f8e2c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e30ff9640 1 -- 172.21.15.62:0/3400319809 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c016420 con 0x7f8e340b2e70 2024-04-18T10:06:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e30ff9640 1 -- 172.21.15.62:0/3400319809 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e2c01b070 con 0x7f8e340b2e70 2024-04-18T10:06:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e3a748640 1 -- 172.21.15.62:0/3400319809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e340b3830 con 0x7f8e340b2e70 2024-04-18T10:06:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e30ff9640 1 -- 172.21.15.62:0/3400319809 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c0167c0 con 0x7f8e340b2e70 2024-04-18T10:06:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e3a748640 1 -- 172.21.15.62:0/3400319809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e341d28b0 con 0x7f8e340b2e70 2024-04-18T10:06:09.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e3a748640 1 -- 172.21.15.62:0/3400319809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e340b95a0 con 0x7f8e340b2e70 2024-04-18T10:06:09.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e30ff9640 1 -- 172.21.15.62:0/3400319809 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8e2c016960 con 0x7f8e340b2e70 2024-04-18T10:06:09.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.842+0000 7f8e30ff9640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8e20068380 0x7f8e2006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:09.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.846+0000 7f8e30ff9640 1 -- 172.21.15.62:0/3400319809 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8e2c0877c0 con 0x7f8e340b2e70 2024-04-18T10:06:09.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.846+0000 7f8e33fff640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8e20068380 0x7f8e2006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:09.844 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.846+0000 7f8e33fff640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8e20068380 0x7f8e2006a840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8e24000c10 tx=0x7f8e24009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:09.847 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:09.850+0000 7f8e30ff9640 1 -- 172.21.15.62:0/3400319809 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8e2c025020 con 0x7f8e340b2e70 2024-04-18T10:06:10.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.042+0000 7f8e3a748640 1 -- 172.21.15.62:0/3400319809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e34068b90 con 0x7f8e340b2e70 2024-04-18T10:06:10.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.042+0000 7f8e30ff9640 1 -- 172.21.15.62:0/3400319809 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3527 (secure 0 0 0) 0x7f8e2c057e10 con 0x7f8e340b2e70 2024-04-18T10:06:10.041 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:10.041 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/904929673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":904929673},{"type":"v1","addr":"172.21.15.145:6835","nonce":904929673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:05:58.588100+0000","modified":"2024-04-18T10:06:01.605447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2120622858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2120622858},{"type":"v1","addr":"172.21.15.62:6833","nonce":2120622858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-18T10:06:10.041 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-04-18T10:06:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.042+0000 7f8e3a748640 1 -- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8e20068380 msgr2=0x7f8e2006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.042+0000 7f8e3a748640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8e20068380 0x7f8e2006a840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8e24000c10 tx=0x7f8e24009040 comp rx=0 tx=0).stop 2024-04-18T10:06:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.042+0000 7f8e3a748640 1 -- 172.21.15.62:0/3400319809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e340b2e70 msgr2=0x7f8e340b32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.042+0000 7f8e3a748640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e340b2e70 0x7f8e340b32f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c009f50 tx=0x7f8e2c00f520 comp rx=0 tx=0).stop 2024-04-18T10:06:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.042+0000 7f8e3a748640 1 -- 172.21.15.62:0/3400319809 shutdown_connections 2024-04-18T10:06:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.042+0000 7f8e3a748640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8e20068380 0x7f8e2006a840 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.042+0000 7f8e3a748640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e340b2e70 0x7f8e340b32f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.042+0000 7f8e3a748640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e341e1380 0x7f8e341e1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.042+0000 7f8e3a748640 1 --2- 172.21.15.62:0/3400319809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e1c004810 0x7f8e341d1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.042+0000 7f8e3a748640 1 -- 172.21.15.62:0/3400319809 >> 172.21.15.62:0/3400319809 conn(0x7f8e3406c0d0 msgr2=0x7f8e340ae980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:10.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.046+0000 7f8e3a748640 1 -- 172.21.15.62:0/3400319809 shutdown_connections 2024-04-18T10:06:10.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.046+0000 7f8e3a748640 1 -- 172.21.15.62:0/3400319809 wait complete. 2024-04-18T10:06:10.059 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:10.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.178+0000 7eff46fa1640 1 Processor -- start 2024-04-18T10:06:10.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.178+0000 7eff46fa1640 1 -- start start 2024-04-18T10:06:10.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff46fa1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff400b44e0 0x7eff400b4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:10.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff46fa1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff40143bb0 0x7eff40144040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:10.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff46fa1640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff400ba170 con 0x7eff400b9d90 2024-04-18T10:06:10.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff46fa1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7eff40145b80 con 0x7eff40143bb0 2024-04-18T10:06:10.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff46fa1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7eff401458a0 con 0x7eff400b44e0 2024-04-18T10:06:10.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff4553e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff40143bb0 0x7eff40144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:10.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff4553e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7eff40143bb0 0x7eff40144040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53620/0 (socket says 172.21.15.62:53620) 2024-04-18T10:06:10.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff4553e640 1 -- 172.21.15.62:0/2970705847 learned_addr learned my addr 172.21.15.62:0/2970705847 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:10.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff3ffff640 1 --2- 172.21.15.62:0/2970705847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff400b44e0 0x7eff400b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:10.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff4553e640 1 -- 172.21.15.62:0/2970705847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff400b44e0 msgr2=0x7eff400b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:10.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff4553e640 1 --2- 172.21.15.62:0/2970705847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff400b44e0 0x7eff400b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff4553e640 1 -- 172.21.15.62:0/2970705847 >> v1:172.21.15.145:6789/0 conn(0x7eff400b9d90 legacy=0x7eff401450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff4553e640 1 -- 172.21.15.62:0/2970705847 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff400b3fc0 con 0x7eff40143bb0 2024-04-18T10:06:10.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff4553e640 1 --2- 172.21.15.62:0/2970705847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff40143bb0 0x7eff40144040 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7eff38009870 tx=0x7eff3801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34438f31adce71b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:10.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff3f7fe640 1 -- 172.21.15.62:0/2970705847 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff380b0070 con 0x7eff40143bb0 2024-04-18T10:06:10.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff3f7fe640 1 -- 172.21.15.62:0/2970705847 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff380ab600 con 0x7eff40143bb0 2024-04-18T10:06:10.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.182+0000 7eff3f7fe640 1 -- 172.21.15.62:0/2970705847 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff380ab9d0 con 0x7eff40143bb0 2024-04-18T10:06:10.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 -- 172.21.15.62:0/2970705847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff40143bb0 msgr2=0x7eff40144040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 --2- 172.21.15.62:0/2970705847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff40143bb0 0x7eff40144040 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7eff38009870 tx=0x7eff3801c4d0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 -- 172.21.15.62:0/2970705847 shutdown_connections 2024-04-18T10:06:10.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 --2- 172.21.15.62:0/2970705847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff40143bb0 0x7eff40144040 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 --2- 172.21.15.62:0/2970705847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff400b44e0 0x7eff400b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 -- 172.21.15.62:0/2970705847 >> 172.21.15.62:0/2970705847 conn(0x7eff4006c0d0 msgr2=0x7eff400b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:10.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 -- 172.21.15.62:0/2970705847 shutdown_connections 2024-04-18T10:06:10.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 -- 172.21.15.62:0/2970705847 wait complete. 2024-04-18T10:06:10.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 Processor -- start 2024-04-18T10:06:10.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 -- start start 2024-04-18T10:06:10.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff400b44e0 0x7eff40138080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:10.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff400b9d90 0x7eff4009c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:10.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff40143bb0 0x7eff4009cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:10.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7eff40152010 con 0x7eff400b9d90 2024-04-18T10:06:10.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7eff40151e90 con 0x7eff40143bb0 2024-04-18T10:06:10.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7eff40152190 con 0x7eff400b44e0 2024-04-18T10:06:10.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff3ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff400b9d90 0x7eff4009c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:10.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff3ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff400b9d90 0x7eff4009c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53626/0 (socket says 172.21.15.62:53626) 2024-04-18T10:06:10.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff3ffff640 1 -- 172.21.15.62:0/3011390812 learned_addr learned my addr 172.21.15.62:0/3011390812 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:10.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff4553e640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff40143bb0 0x7eff4009cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:10.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff44d3d640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff400b44e0 0x7eff40138080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:10.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff3ffff640 1 -- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff400b44e0 msgr2=0x7eff40138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff3ffff640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff400b44e0 0x7eff40138080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff3ffff640 1 -- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff40143bb0 msgr2=0x7eff4009cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff3ffff640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff40143bb0 0x7eff4009cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff3ffff640 1 -- 172.21.15.62:0/3011390812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff400b79b0 con 0x7eff400b9d90 2024-04-18T10:06:10.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff3ffff640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff400b9d90 0x7eff4009c6f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7eff30007f20 tx=0x7eff3000e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:10.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff3d7fa640 1 -- 172.21.15.62:0/3011390812 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30014620 con 0x7eff400b9d90 2024-04-18T10:06:10.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff3d7fa640 1 -- 172.21.15.62:0/3011390812 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff30019070 con 0x7eff400b9d90 2024-04-18T10:06:10.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 -- 172.21.15.62:0/3011390812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff400b8fb0 con 0x7eff400b9d90 2024-04-18T10:06:10.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff3d7fa640 1 -- 172.21.15.62:0/3011390812 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff30014a00 con 0x7eff400b9d90 2024-04-18T10:06:10.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 -- 172.21.15.62:0/3011390812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff401e0b70 con 0x7eff400b9d90 2024-04-18T10:06:10.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.186+0000 7eff46fa1640 1 -- 172.21.15.62:0/3011390812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff40068b90 con 0x7eff400b9d90 2024-04-18T10:06:10.188 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.190+0000 7eff3d7fa640 1 -- 172.21.15.62:0/3011390812 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7eff30014ba0 con 0x7eff400b9d90 2024-04-18T10:06:10.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.190+0000 7eff3d7fa640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7eff20068380 0x7eff2006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:10.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.190+0000 7eff3d7fa640 1 -- 172.21.15.62:0/3011390812 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7eff30087250 con 0x7eff400b9d90 2024-04-18T10:06:10.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.190+0000 7eff44d3d640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7eff20068380 0x7eff2006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:10.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.190+0000 7eff44d3d640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7eff20068380 0x7eff2006a840 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7eff28001650 tx=0x7eff28009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:10.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.194+0000 7eff3d7fa640 1 -- 172.21.15.62:0/3011390812 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7eff300578a0 con 0x7eff400b9d90 2024-04-18T10:06:10.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.358+0000 7eff46fa1640 1 -- 172.21.15.62:0/3011390812 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff401393f0 con 0x7eff400b9d90 2024-04-18T10:06:10.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff3d7fa640 1 -- 172.21.15.62:0/3011390812 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7eff300571a0 con 0x7eff400b9d90 2024-04-18T10:06:10.360 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:10.360 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:10.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff46fa1640 1 -- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7eff20068380 msgr2=0x7eff2006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff46fa1640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7eff20068380 0x7eff2006a840 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7eff28001650 tx=0x7eff28009040 comp rx=0 tx=0).stop 2024-04-18T10:06:10.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff46fa1640 1 -- 172.21.15.62:0/3011390812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff400b9d90 msgr2=0x7eff4009c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff46fa1640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff400b9d90 0x7eff4009c6f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7eff30007f20 tx=0x7eff3000e570 comp rx=0 tx=0).stop 2024-04-18T10:06:10.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff46fa1640 1 -- 172.21.15.62:0/3011390812 shutdown_connections 2024-04-18T10:06:10.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff46fa1640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7eff20068380 0x7eff2006a840 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff46fa1640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff40143bb0 0x7eff4009cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff46fa1640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff400b9d90 0x7eff4009c6f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff46fa1640 1 --2- 172.21.15.62:0/3011390812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff400b44e0 0x7eff40138080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff46fa1640 1 -- 172.21.15.62:0/3011390812 >> 172.21.15.62:0/3011390812 conn(0x7eff4006c0d0 msgr2=0x7eff401409b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:10.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff46fa1640 1 -- 172.21.15.62:0/3011390812 shutdown_connections 2024-04-18T10:06:10.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.362+0000 7eff46fa1640 1 -- 172.21.15.62:0/3011390812 wait complete. 2024-04-18T10:06:10.379 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:10.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa5d5c1640 1 Processor -- start 2024-04-18T10:06:10.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa5d5c1640 1 -- start start 2024-04-18T10:06:10.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa5d5c1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffa580b4f80 0x7ffa580b7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:10.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa5d5c1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa58071220 con 0x7ffa58070e40 2024-04-18T10:06:10.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa5d5c1640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa580b3cb0 con 0x7ffa5814e370 2024-04-18T10:06:10.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa5d5c1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7ffa580b4af0 con 0x7ffa580b4f80 2024-04-18T10:06:10.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa567fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffa580b4f80 0x7ffa580b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa567fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ffa580b4f80 0x7ffa580b7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53638/0 (socket says 172.21.15.62:53638) 2024-04-18T10:06:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa567fc640 1 -- 172.21.15.62:0/463035126 learned_addr learned my addr 172.21.15.62:0/463035126 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa56ffd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ffa58070e40 0x7ffa58142d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:55044/0 (socket says 172.21.15.62:55044) 2024-04-18T10:06:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa567fc640 1 -- 172.21.15.62:0/463035126 >> v1:172.21.15.145:6789/0 conn(0x7ffa5814e370 legacy=0x7ffa58150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa567fc640 1 -- 172.21.15.62:0/463035126 >> v1:172.21.15.62:6789/0 conn(0x7ffa58070e40 legacy=0x7ffa58142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa55ffb640 1 -- 172.21.15.62:0/463035126 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2286684509 0 0) 0x7ffa58071220 con 0x7ffa58070e40 2024-04-18T10:06:10.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.498+0000 7ffa567fc640 1 -- 172.21.15.62:0/463035126 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa580b3f90 con 0x7ffa580b4f80 2024-04-18T10:06:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.502+0000 7ffa567fc640 1 --2- 172.21.15.62:0/463035126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa580b4f80 0x7ffa580b7150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009e50 tx=0x7ffa4801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3d7b7f75f8a102 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.502+0000 7ffa55ffb640 1 -- 172.21.15.62:0/463035126 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa480a56e0 con 0x7ffa580b4f80 2024-04-18T10:06:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.502+0000 7ffa55ffb640 1 -- 172.21.15.62:0/463035126 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa480a5880 con 0x7ffa580b4f80 2024-04-18T10:06:10.501 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.502+0000 7ffa55ffb640 1 -- 172.21.15.62:0/463035126 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa480a5c50 con 0x7ffa580b4f80 2024-04-18T10:06:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.502+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/463035126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa580b4f80 msgr2=0x7ffa580b7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.502+0000 7ffa5d5c1640 1 --2- 172.21.15.62:0/463035126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa580b4f80 0x7ffa580b7150 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009e50 tx=0x7ffa4801c910 comp rx=0 tx=0).stop 2024-04-18T10:06:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.502+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/463035126 shutdown_connections 2024-04-18T10:06:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.502+0000 7ffa5d5c1640 1 --2- 172.21.15.62:0/463035126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa580b4f80 0x7ffa580b7150 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.502+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/463035126 >> 172.21.15.62:0/463035126 conn(0x7ffa580b8240 msgr2=0x7ffa580b8650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.502+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/463035126 shutdown_connections 2024-04-18T10:06:10.502 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.502+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/463035126 wait complete. 2024-04-18T10:06:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa5d5c1640 1 Processor -- start 2024-04-18T10:06:10.503 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa5d5c1640 1 -- start start 2024-04-18T10:06:10.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa5d5c1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa58070e40 0x7ffa581def50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:10.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa5d5c1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa580b4f80 0x7ffa5813d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa5d5c1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5814e370 0x7ffa581e1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa5d5c1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ffa580b2e50 con 0x7ffa580b4f80 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa5d5c1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7ffa580b3930 con 0x7ffa58070e40 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa5d5c1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7ffa581546a0 con 0x7ffa5814e370 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa567fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa580b4f80 0x7ffa5813d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa567fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa580b4f80 0x7ffa5813d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:53642/0 (socket says 172.21.15.62:53642) 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa567fc640 1 -- 172.21.15.62:0/1698275597 learned_addr learned my addr 172.21.15.62:0/1698275597 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa56ffd640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa58070e40 0x7ffa581def50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa577fe640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5814e370 0x7ffa581e1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa567fc640 1 -- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5814e370 msgr2=0x7ffa581e1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa567fc640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5814e370 0x7ffa581e1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa567fc640 1 -- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa58070e40 msgr2=0x7ffa581def50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa567fc640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa58070e40 0x7ffa581def50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa567fc640 1 -- 172.21.15.62:0/1698275597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa581de790 con 0x7ffa580b4f80 2024-04-18T10:06:10.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa56ffd640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa58070e40 0x7ffa581def50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:10.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa567fc640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa580b4f80 0x7ffa5813d5b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffa480af040 tx=0x7ffa480196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:10.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa4f7fe640 1 -- 172.21.15.62:0/1698275597 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa480bc020 con 0x7ffa580b4f80 2024-04-18T10:06:10.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa4f7fe640 1 -- 172.21.15.62:0/1698275597 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa480b8690 con 0x7ffa580b4f80 2024-04-18T10:06:10.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/1698275597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa5813eca0 con 0x7ffa580b4f80 2024-04-18T10:06:10.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa4f7fe640 1 -- 172.21.15.62:0/1698275597 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa480b8a80 con 0x7ffa580b4f80 2024-04-18T10:06:10.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/1698275597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa5813bc00 con 0x7ffa580b4f80 2024-04-18T10:06:10.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/1698275597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa58069230 con 0x7ffa580b4f80 2024-04-18T10:06:10.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.506+0000 7ffa4f7fe640 1 -- 172.21.15.62:0/1698275597 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffa480bf020 con 0x7ffa580b4f80 2024-04-18T10:06:10.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.510+0000 7ffa4f7fe640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ffa38068380 0x7ffa3806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:10.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.510+0000 7ffa4f7fe640 1 -- 172.21.15.62:0/1698275597 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ffa48121960 con 0x7ffa580b4f80 2024-04-18T10:06:10.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.510+0000 7ffa56ffd640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ffa38068380 0x7ffa3806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:10.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.510+0000 7ffa56ffd640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ffa38068380 0x7ffa3806a840 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffa50000f70 tx=0x7ffa50007dd0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:10.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.514+0000 7ffa4f7fe640 1 -- 172.21.15.62:0/1698275597 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ffa480c6030 con 0x7ffa580b4f80 2024-04-18T10:06:10.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.714+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/1698275597 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa580b90e0 con 0x7ffa580b4f80 2024-04-18T10:06:10.714 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.714+0000 7ffa4f7fe640 1 -- 172.21.15.62:0/1698275597 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3527 (secure 0 0 0) 0x7ffa48016620 con 0x7ffa580b4f80 2024-04-18T10:06:10.715 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:10.715 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/904929673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":904929673},{"type":"v1","addr":"172.21.15.145:6835","nonce":904929673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:05:58.588100+0000","modified":"2024-04-18T10:06:01.605447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2120622858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2120622858},{"type":"v1","addr":"172.21.15.62:6833","nonce":2120622858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-18T10:06:10.715 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-04-18T10:06:10.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.718+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ffa38068380 msgr2=0x7ffa3806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.718+0000 7ffa5d5c1640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ffa38068380 0x7ffa3806a840 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffa50000f70 tx=0x7ffa50007dd0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.718+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/1698275597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa580b4f80 msgr2=0x7ffa5813d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:10.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.718+0000 7ffa5d5c1640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa580b4f80 0x7ffa5813d5b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffa480af040 tx=0x7ffa480196b0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.718+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/1698275597 shutdown_connections 2024-04-18T10:06:10.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.718+0000 7ffa5d5c1640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ffa38068380 0x7ffa3806a840 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.718+0000 7ffa5d5c1640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa5814e370 0x7ffa581e1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.718+0000 7ffa5d5c1640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa580b4f80 0x7ffa5813d5b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.718+0000 7ffa5d5c1640 1 --2- 172.21.15.62:0/1698275597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa58070e40 0x7ffa581def50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:10.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.718+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/1698275597 >> 172.21.15.62:0/1698275597 conn(0x7ffa580b8240 msgr2=0x7ffa58074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:10.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.718+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/1698275597 shutdown_connections 2024-04-18T10:06:10.718 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:10.718+0000 7ffa5d5c1640 1 -- 172.21.15.62:0/1698275597 wait complete. 2024-04-18T10:06:10.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-18T10:05:58.588100+0000', 'modified': '2024-04-18T10:06:01.605447+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4251}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4251': {'gid': 4251, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.62:6833/2120622858', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 2120622858}, {'type': 'v1', 'addr': '172.21.15.62:6833', 'nonce': 2120622858}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4251, 'qdb_cluster': [4251]} 2024-04-18T10:06:10.935 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-18T10:06:10.936 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-18T10:06:11.069 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f273b916640 1 Processor -- start 2024-04-18T10:06:11.070 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f273b916640 1 -- start start 2024-04-18T10:06:11.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f273b916640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f273409c4d0 0x7f273409ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f273b916640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f273409e2d0 0x7f2734142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f273b916640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f273409d430 con 0x7f273409b620 2024-04-18T10:06:11.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f273b916640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f273406eb30 con 0x7f273409e2d0 2024-04-18T10:06:11.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f273b916640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f273409dea0 con 0x7f273409c4d0 2024-04-18T10:06:11.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f2739eb3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f273409e2d0 0x7f2734142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.071 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f2739eb3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f273409e2d0 0x7f2734142ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41718/0 (socket says 172.21.15.62:41718) 2024-04-18T10:06:11.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f2739eb3640 1 -- 172.21.15.62:0/2845105777 learned_addr learned my addr 172.21.15.62:0/2845105777 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:11.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f2738eb1640 1 --2- 172.21.15.62:0/2845105777 >> v2:172.21.15.145:3300/0 conn(0x7f273409c4d0 0x7f273409ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f2739eb3640 1 -- 172.21.15.62:0/2845105777 >> v2:172.21.15.145:3300/0 conn(0x7f273409c4d0 msgr2=0x7f273409ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f2739eb3640 1 --2- 172.21.15.62:0/2845105777 >> v2:172.21.15.145:3300/0 conn(0x7f273409c4d0 0x7f273409ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f2739eb3640 1 -- 172.21.15.62:0/2845105777 >> v1:172.21.15.145:6789/0 conn(0x7f273409b620 legacy=0x7f27341492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.072 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.070+0000 7f2739eb3640 1 -- 172.21.15.62:0/2845105777 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273409d6a0 con 0x7f273409e2d0 2024-04-18T10:06:11.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.074+0000 7f2739eb3640 1 --2- 172.21.15.62:0/2845105777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f273409e2d0 0x7f2734142ba0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f272c009e50 tx=0x7f272c01c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc8f816e70ee146 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:11.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.074+0000 7f272bfff640 1 -- 172.21.15.62:0/2845105777 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272c0ac020 con 0x7f273409e2d0 2024-04-18T10:06:11.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.074+0000 7f272bfff640 1 -- 172.21.15.62:0/2845105777 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f272c0a6420 con 0x7f273409e2d0 2024-04-18T10:06:11.073 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.074+0000 7f272bfff640 1 -- 172.21.15.62:0/2845105777 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272c0a67f0 con 0x7f273409e2d0 2024-04-18T10:06:11.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.074+0000 7f273b916640 1 -- 172.21.15.62:0/2845105777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f273409e2d0 msgr2=0x7f2734142ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.074+0000 7f273b916640 1 --2- 172.21.15.62:0/2845105777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f273409e2d0 0x7f2734142ba0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f272c009e50 tx=0x7f272c01c660 comp rx=0 tx=0).stop 2024-04-18T10:06:11.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.074+0000 7f273b916640 1 -- 172.21.15.62:0/2845105777 shutdown_connections 2024-04-18T10:06:11.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.074+0000 7f273b916640 1 --2- 172.21.15.62:0/2845105777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f273409e2d0 0x7f2734142ba0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.074+0000 7f273b916640 1 --2- 172.21.15.62:0/2845105777 >> v2:172.21.15.145:3300/0 conn(0x7f273409c4d0 0x7f273409ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.074+0000 7f273b916640 1 -- 172.21.15.62:0/2845105777 >> 172.21.15.62:0/2845105777 conn(0x7f27340b9420 msgr2=0x7f27340b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:11.074 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.074+0000 7f273b916640 1 -- 172.21.15.62:0/2845105777 shutdown_connections 2024-04-18T10:06:11.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.074+0000 7f273b916640 1 -- 172.21.15.62:0/2845105777 wait complete. 2024-04-18T10:06:11.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f273b916640 1 Processor -- start 2024-04-18T10:06:11.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f273b916640 1 -- start start 2024-04-18T10:06:11.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f273b916640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f273409b620 0x7f27340b7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f273b916640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f273409c4d0 0x7f27341e3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f273b916640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f273409e2d0 0x7f27341e4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f273b916640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2734152b90 con 0x7f273409b620 2024-04-18T10:06:11.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f273b916640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f2734152a10 con 0x7f273409c4d0 2024-04-18T10:06:11.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f273b916640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2734152d10 con 0x7f273409e2d0 2024-04-18T10:06:11.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f27396b2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f273409b620 0x7f27340b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f27396b2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f273409b620 0x7f27340b7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41730/0 (socket says 172.21.15.62:41730) 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f27396b2640 1 -- 172.21.15.62:0/1352959387 learned_addr learned my addr 172.21.15.62:0/1352959387 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f27396b2640 1 -- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f273409e2d0 msgr2=0x7f27341e4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f2739eb3640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f273409e2d0 0x7f27341e4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f2738eb1640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f273409c4d0 0x7f27341e3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f27396b2640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f273409e2d0 0x7f27341e4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f27396b2640 1 -- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f273409c4d0 msgr2=0x7f27341e3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f27396b2640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f273409c4d0 0x7f27341e3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f27396b2640 1 -- 172.21.15.62:0/1352959387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27340b8590 con 0x7f273409b620 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f27396b2640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f273409b620 0x7f27340b7480 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f273000c450 tx=0x7f273000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f2729ffb640 1 -- 172.21.15.62:0/1352959387 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730011790 con 0x7f273409b620 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f2729ffb640 1 -- 172.21.15.62:0/1352959387 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2730017070 con 0x7f273409b620 2024-04-18T10:06:11.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f273b916640 1 -- 172.21.15.62:0/1352959387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27340b8870 con 0x7f273409b620 2024-04-18T10:06:11.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f2729ffb640 1 -- 172.21.15.62:0/1352959387 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730011b90 con 0x7f273409b620 2024-04-18T10:06:11.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f273b916640 1 -- 172.21.15.62:0/1352959387 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f273413f6c0 con 0x7f273409b620 2024-04-18T10:06:11.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f2729ffb640 1 -- 172.21.15.62:0/1352959387 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2730011d30 con 0x7f273409b620 2024-04-18T10:06:11.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f2729ffb640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2718068380 0x7f271806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f2729ffb640 1 -- 172.21.15.62:0/1352959387 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f27300867a0 con 0x7f273409b620 2024-04-18T10:06:11.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.078+0000 7f2738eb1640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2718068380 0x7f271806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 Processor -- start 2024-04-18T10:06:11.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- start start 2024-04-18T10:06:11.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f2738eb1640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2718068380 0x7f271806a840 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f27340b84c0 tx=0x7f2724008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:11.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580bfc80 0x55a8580c0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 0x55a8580ea9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580eaf10 0x55a8580eb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a857fa16f0 con 0x55a8580c05a0 2024-04-18T10:06:11.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x55a857f084d0 con 0x55a8580eaf10 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x55a857ea2bf0 con 0x55a8580bfc80 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f2738eb1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 0x55a8580ea9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f2738eb1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 0x55a8580ea9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41738/0 (socket says 172.21.15.62:41738) 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f2738eb1640 1 -- 172.21.15.62:0/2550075635 learned_addr learned my addr 172.21.15.62:0/2550075635 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f27396b2640 1 --2- 172.21.15.62:0/2550075635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580bfc80 0x55a8580c0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f2738eb1640 1 -- 172.21.15.62:0/2550075635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580bfc80 msgr2=0x55a8580c0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f2738eb1640 1 --2- 172.21.15.62:0/2550075635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580bfc80 0x55a8580c0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f2738eb1640 1 -- 172.21.15.62:0/2550075635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580eaf10 msgr2=0x55a8580eb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f2738eb1640 1 --2- 172.21.15.62:0/2550075635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580eaf10 0x55a8580eb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f2738eb1640 1 -- 172.21.15.62:0/2550075635 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a857ea2e40 con 0x55a8580c05a0 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f2738eb1640 1 --2- 172.21.15.62:0/2550075635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 0x55a8580ea9d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f272400e810 tx=0x7f2724011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff71d3e116893bb server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f271effd640 1 -- 172.21.15.62:0/2550075635 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2724025070 con 0x55a8580c05a0 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f271effd640 1 -- 172.21.15.62:0/2550075635 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2724015640 con 0x55a8580c05a0 2024-04-18T10:06:11.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f271effd640 1 -- 172.21.15.62:0/2550075635 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272400f040 con 0x55a8580c05a0 2024-04-18T10:06:11.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- 172.21.15.62:0/2550075635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 msgr2=0x55a8580ea9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 --2- 172.21.15.62:0/2550075635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 0x55a8580ea9d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f272400e810 tx=0x7f2724011000 comp rx=0 tx=0).stop 2024-04-18T10:06:11.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- 172.21.15.62:0/2550075635 shutdown_connections 2024-04-18T10:06:11.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 --2- 172.21.15.62:0/2550075635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580eaf10 0x55a8580eb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 --2- 172.21.15.62:0/2550075635 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 0x55a8580ea9d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 --2- 172.21.15.62:0/2550075635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580bfc80 0x55a8580c0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- 172.21.15.62:0/2550075635 >> 172.21.15.62:0/2550075635 conn(0x55a8580e6780 msgr2=0x55a8580e8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:11.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- 172.21.15.62:0/2550075635 shutdown_connections 2024-04-18T10:06:11.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- 172.21.15.62:0/2550075635 wait complete. 2024-04-18T10:06:11.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- start start 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 Processor -- start 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580bfc80 0x55a8580bf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580c05a0 0x55a8580f0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580eaf10 0x55a8580f1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a857fc4e50 con 0x55a8580bfc80 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x55a857ec5a60 con 0x55a8580c05a0 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x55a857ec3cf0 con 0x55a8580eaf10 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f273ea98000 10 client.0 _ll_register_callbacks cb 0x55a8580a5a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.082+0000 7f27396b2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580bfc80 0x55a8580bf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f27396b2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580bfc80 0x55a8580bf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41746/0 (socket says 172.21.15.62:41746) 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f27396b2640 1 -- 172.21.15.62:0/3781796817 learned_addr learned my addr 172.21.15.62:0/3781796817 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2738eb1640 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580c05a0 0x55a8580f0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2739eb3640 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580eaf10 0x55a8580f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f27396b2640 1 -- 172.21.15.62:0/3781796817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580eaf10 msgr2=0x55a8580f1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f27396b2640 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580eaf10 0x55a8580f1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f27396b2640 1 -- 172.21.15.62:0/3781796817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580c05a0 msgr2=0x55a8580f0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f27396b2640 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580c05a0 0x55a8580f0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f27396b2640 1 -- 172.21.15.62:0/3781796817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a857ed3e00 con 0x55a8580bfc80 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2739eb3640 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580eaf10 0x55a8580f1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f27396b2640 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580bfc80 0x55a8580bf3c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2730084d80 tx=0x7f2730056df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2e9eb134af24d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2713fff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2713fff640 1 -- 172.21.15.62:0/3781796817 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730083060 con 0x55a8580bfc80 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2713fff640 1 -- 172.21.15.62:0/3781796817 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f27300848b0 con 0x55a8580bfc80 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 -- 172.21.15.62:0/3781796817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2724015640 con 0x55a8580bfc80 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2713fff640 1 -- 172.21.15.62:0/3781796817 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f2730073050 con 0x55a8580bfc80 2024-04-18T10:06:11.090 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 10 client.4602 fetch_fsmap learned FSMap version 8 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 -- 172.21.15.62:0/3781796817 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55a857f8a670 con 0x55a8580bfc80 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2713fff640 1 -- 172.21.15.62:0/3781796817 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2724015640 con 0x55a8580bfc80 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2713fff640 1 -- 172.21.15.62:0/3781796817 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 8) ==== 2284+0+0 (secure 0 0 0) 0x7f273005ee40 con 0x55a8580bfc80 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 10 client.4602 fetch_fsmap finished waiting for FSMap version 8 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 10 client.4602 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 20 client.4602 populate_metadata read hostname 'smithi062' 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.62:6832/2120622858,v1:172.21.15.62:6833/2120622858] conn(0x55a8580f1a10 0x55a8580f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 4 client.4602 mds_command: new command op to 4251 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 -- 172.21.15.62:0/3781796817 --> [v2:172.21.15.62:6832/2120622858,v1:172.21.15.62:6833/2120622858] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a857ea1bc0 con 0x55a8580f1a10 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2738eb1640 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.62:6832/2120622858,v1:172.21.15.62:6833/2120622858] conn(0x55a8580f1a10 0x55a8580f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2738eb1640 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.62:6832/2120622858,v1:172.21.15.62:6833/2120622858] conn(0x55a8580f1a10 0x55a8580f1df0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=382993dd82afd36f server_cookie=43402ede488a71c0 in_seq=0 out_seq=0 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2713fff640 10 client.4602 ms_handle_connect on v2:172.21.15.62:6832/2120622858 2024-04-18T10:06:11.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2713fff640 1 -- 172.21.15.62:0/3781796817 <== mds.0 v2:172.21.15.62:6832/2120622858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a857ea1bc0 con 0x55a8580f1a10 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2713fff640 10 client.4602 handle_command_reply: tid=0 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 client.4602 shutdown 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 -- 172.21.15.62:0/3781796817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580bfc80 msgr2=0x55a8580bf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580bfc80 0x55a8580bf3c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2730084d80 tx=0x7f2730056df0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 -- 172.21.15.62:0/3781796817 shutdown_connections 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.62:6832/2120622858,v1:172.21.15.62:6833/2120622858] conn(0x55a8580f1a10 0x55a8580f1df0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580eaf10 0x55a8580f1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f2713fff640 0 client.4602 ms_handle_reset on v2:172.21.15.62:6832/2120622858 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580c05a0 0x55a8580f0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 --2- 172.21.15.62:0/3781796817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580bfc80 0x55a8580bf3c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 -- 172.21.15.62:0/3781796817 >> 172.21.15.62:0/3781796817 conn(0x55a8580e6780 msgr2=0x55a8580e8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 -- 172.21.15.62:0/3781796817 shutdown_connections 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 1 -- 172.21.15.62:0/3781796817 wait complete. 2024-04-18T10:06:11.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.086+0000 7f273ea98000 20 client.4602 trim_cache size 0 max 16384 2024-04-18T10:06:11.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 Processor -- start 2024-04-18T10:06:11.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 -- start start 2024-04-18T10:06:11.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580bfc80 0x55a8581062b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580c05a0 0x55a8580bef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580eaf10 0x55a8580bf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f27341ed0e0 con 0x55a8580eaf10 2024-04-18T10:06:11.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f27341ec270 con 0x55a8580bfc80 2024-04-18T10:06:11.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x55a857f077e0 con 0x55a8580c05a0 2024-04-18T10:06:11.111 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f2739eb3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580eaf10 0x55a8580bf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f2739eb3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580eaf10 0x55a8580bf4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41748/0 (socket says 172.21.15.62:41748) 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f2739eb3640 1 -- 172.21.15.62:0/1436133470 learned_addr learned my addr 172.21.15.62:0/1436133470 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f27396b2640 1 --2- 172.21.15.62:0/1436133470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580bfc80 0x55a8581062b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f2738eb1640 1 --2- 172.21.15.62:0/1436133470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580c05a0 0x55a8580bef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f2739eb3640 1 -- 172.21.15.62:0/1436133470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580c05a0 msgr2=0x55a8580bef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f2739eb3640 1 --2- 172.21.15.62:0/1436133470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580c05a0 0x55a8580bef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f2739eb3640 1 -- 172.21.15.62:0/1436133470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580bfc80 msgr2=0x55a8581062b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f2739eb3640 1 --2- 172.21.15.62:0/1436133470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580bfc80 0x55a8581062b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f2739eb3640 1 -- 172.21.15.62:0/1436133470 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a857f61340 con 0x55a8580eaf10 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f2738eb1640 1 --2- 172.21.15.62:0/1436133470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580c05a0 0x55a8580bef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f2739eb3640 1 --2- 172.21.15.62:0/1436133470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580eaf10 0x55a8580bf4c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f272c0ab040 tx=0x7f272c004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670c4b891438f17 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f271e7fc640 1 -- 172.21.15.62:0/1436133470 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272c0bf070 con 0x55a8580eaf10 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f271e7fc640 1 -- 172.21.15.62:0/1436133470 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f272c009190 con 0x55a8580eaf10 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f271e7fc640 1 -- 172.21.15.62:0/1436133470 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272c0b1660 con 0x55a8580eaf10 2024-04-18T10:06:11.112 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 -- 172.21.15.62:0/1436133470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580eaf10 msgr2=0x55a8580bf4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 --2- 172.21.15.62:0/1436133470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580eaf10 0x55a8580bf4c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f272c0ab040 tx=0x7f272c004ac0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 -- 172.21.15.62:0/1436133470 shutdown_connections 2024-04-18T10:06:11.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 --2- 172.21.15.62:0/1436133470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580eaf10 0x55a8580bf4c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 --2- 172.21.15.62:0/1436133470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580c05a0 0x55a8580bef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 --2- 172.21.15.62:0/1436133470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580bfc80 0x55a8581062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 -- 172.21.15.62:0/1436133470 >> 172.21.15.62:0/1436133470 conn(0x55a8580e6780 msgr2=0x55a8580fbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:11.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 -- 172.21.15.62:0/1436133470 shutdown_connections 2024-04-18T10:06:11.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 -- 172.21.15.62:0/1436133470 wait complete. 2024-04-18T10:06:11.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 -- start start 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.106+0000 7f273ea98000 1 Processor -- start 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580bfc80 0x55a8580bf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 0x55a8580be410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580eaf10 0x55a8580bc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x55a857f156e0 con 0x55a8580c05a0 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x55a857f0cb00 con 0x55a8580eaf10 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x55a857e9e7b0 con 0x55a8580bfc80 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 10 client.0 _ll_register_callbacks cb 0x7f273005d0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2738eb1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 0x55a8580be410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2738eb1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 0x55a8580be410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41758/0 (socket says 172.21.15.62:41758) 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2738eb1640 1 -- 172.21.15.62:0/1087942690 learned_addr learned my addr 172.21.15.62:0/1087942690 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f27396b2640 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580bfc80 0x55a8580bf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2739eb3640 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580eaf10 0x55a8580bc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2738eb1640 1 -- 172.21.15.62:0/1087942690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580bfc80 msgr2=0x55a8580bf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2738eb1640 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580bfc80 0x55a8580bf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2738eb1640 1 -- 172.21.15.62:0/1087942690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580eaf10 msgr2=0x55a8580bc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2738eb1640 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580eaf10 0x55a8580bc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2738eb1640 1 -- 172.21.15.62:0/1087942690 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a857e9a7c0 con 0x55a8580c05a0 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2738eb1640 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 0x55a8580be410 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2724008010 tx=0x7f2724004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ea191d1885bf78 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2713fff640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2713fff640 1 -- 172.21.15.62:0/1087942690 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2724025040 con 0x55a8580c05a0 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2713fff640 1 -- 172.21.15.62:0/1087942690 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2724015a10 con 0x55a8580c05a0 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 -- 172.21.15.62:0/1087942690 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f272c009190 con 0x55a8580c05a0 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2713fff640 1 -- 172.21.15.62:0/1087942690 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7f2724042050 con 0x55a8580c05a0 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 10 client.4608 fetch_fsmap learned FSMap version 8 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 -- 172.21.15.62:0/1087942690 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x55a857ead650 con 0x55a8580c05a0 2024-04-18T10:06:11.115 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2713fff640 1 -- 172.21.15.62:0/1087942690 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f272c009190 con 0x55a8580c05a0 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2713fff640 1 -- 172.21.15.62:0/1087942690 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 8) ==== 2284+0+0 (secure 0 0 0) 0x7f2724045020 con 0x55a8580c05a0 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 10 client.4608 fetch_fsmap finished waiting for FSMap version 8 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 10 client.4608 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 20 client.4608 populate_metadata read hostname 'smithi062' 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.62:6832/2120622858,v1:172.21.15.62:6833/2120622858] conn(0x55a8580bc930 0x55a8580bcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 4 client.4608 mds_command: new command op to 4251 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 -- 172.21.15.62:0/1087942690 --> [v2:172.21.15.62:6832/2120622858,v1:172.21.15.62:6833/2120622858] -- command(tid 0: {"prefix": "status"}) -- 0x55a857edd4e0 con 0x55a8580bc930 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f27396b2640 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.62:6832/2120622858,v1:172.21.15.62:6833/2120622858] conn(0x55a8580bc930 0x55a8580bcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2713fff640 10 client.4608 ms_handle_connect on v2:172.21.15.62:6832/2120622858 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f27396b2640 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.62:6832/2120622858,v1:172.21.15.62:6833/2120622858] conn(0x55a8580bc930 0x55a8580bcd10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c4758e90f204331 server_cookie=55d6df98c4894be0 in_seq=0 out_seq=0 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2713fff640 1 -- 172.21.15.62:0/1087942690 <== mds.0 v2:172.21.15.62:6832/2120622858 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55a857edd4e0 con 0x55a8580bc930 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2713fff640 10 client.4608 handle_command_reply: tid=0 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 client.4608 shutdown 2024-04-18T10:06:11.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 -- 172.21.15.62:0/1087942690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 msgr2=0x55a8580be410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 0x55a8580be410 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2724008010 tx=0x7f2724004740 comp rx=0 tx=0).stop 2024-04-18T10:06:11.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 -- 172.21.15.62:0/1087942690 shutdown_connections 2024-04-18T10:06:11.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.62:6832/2120622858,v1:172.21.15.62:6833/2120622858] conn(0x55a8580bc930 0x55a8580bcd10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a8580eaf10 0x55a8580bc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a8580c05a0 0x55a8580be410 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 --2- 172.21.15.62:0/1087942690 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a8580bfc80 0x55a8580bf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 -- 172.21.15.62:0/1087942690 >> 172.21.15.62:0/1087942690 conn(0x55a8580e6780 msgr2=0x55a8581052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:11.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f2713fff640 0 client.4608 ms_handle_reset on v2:172.21.15.62:6832/2120622858 2024-04-18T10:06:11.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 -- 172.21.15.62:0/1087942690 shutdown_connections 2024-04-18T10:06:11.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 1 -- 172.21.15.62:0/1087942690 wait complete. 2024-04-18T10:06:11.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273ea98000 20 client.4608 trim_cache size 0 max 16384 2024-04-18T10:06:11.118 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:06:11.118 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "26864971-6592-407b-b492-2c0e3b7ea567", 2024-04-18T10:06:11.118 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-04-18T10:06:11.118 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 4251, 2024-04-18T10:06:11.118 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-04-18T10:06:11.118 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-04-18T10:06:11.118 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-04-18T10:06:11.118 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 11.892807834999999, 2024-04-18T10:06:11.118 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 8, 2024-04-18T10:06:11.119 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 21, 2024-04-18T10:06:11.119 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 21, 2024-04-18T10:06:11.119 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 18.857480682999999 2024-04-18T10:06:11.119 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:06:11.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273b916640 1 -- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2718068380 msgr2=0x7f271806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273b916640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2718068380 0x7f271806a840 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f27340b84c0 tx=0x7f2724008040 comp rx=0 tx=0).stop 2024-04-18T10:06:11.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273b916640 1 -- 172.21.15.62:0/1352959387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f273409b620 msgr2=0x7f27340b7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273b916640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f273409b620 0x7f27340b7480 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f273000c450 tx=0x7f273000c910 comp rx=0 tx=0).stop 2024-04-18T10:06:11.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273b916640 1 -- 172.21.15.62:0/1352959387 shutdown_connections 2024-04-18T10:06:11.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273b916640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2718068380 0x7f271806a840 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273b916640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f273409e2d0 0x7f27341e4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273b916640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f273409c4d0 0x7f27341e3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273b916640 1 --2- 172.21.15.62:0/1352959387 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f273409b620 0x7f27340b7480 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273b916640 1 -- 172.21.15.62:0/1352959387 >> 172.21.15.62:0/1352959387 conn(0x7f27340b9420 msgr2=0x7f2734145db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:11.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273b916640 1 -- 172.21.15.62:0/1352959387 shutdown_connections 2024-04-18T10:06:11.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.110+0000 7f273b916640 1 -- 172.21.15.62:0/1352959387 wait complete. 2024-04-18T10:06:11.127 INFO:teuthology.run_tasks:Running task sequential... 2024-04-18T10:06:11.136 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-18T10:06:11.136 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-18T10:06:11.137 INFO:teuthology.task.print:Enabling mgr modules 2024-04-18T10:06:11.137 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-18T10:06:11.174 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-18T10:06:11.174 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-04-18T10:06:11.174 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-18T10:06:11.174 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-18T10:06:11.175 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:11.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:11.175 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-04-18T10:06:11.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:11.181 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link delete ceph-brx 2024-04-18T10:06:11.237 INFO:teuthology.orchestra.run.smithi062.stderr:Cannot find device "ceph-brx" 2024-04-18T10:06:11.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T10:06:11.239 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-18T10:06:11.239 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-18T10:06:11.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6900b640 1 Processor -- start 2024-04-18T10:06:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6900b640 1 -- start start 2024-04-18T10:06:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6900b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0c640b44e0 0x7f0c640b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6900b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c640ba170 con 0x7f0c64143bb0 2024-04-18T10:06:11.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6900b640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c641458a0 con 0x7f0c640b9d90 2024-04-18T10:06:11.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6900b640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f0c64145b80 con 0x7f0c640b44e0 2024-04-18T10:06:11.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6359e640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f0c64143bb0 0x7f0c64144020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37062/0 (socket says 172.21.15.62:37062) 2024-04-18T10:06:11.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6359e640 1 -- 172.21.15.62:0/1134899996 learned_addr learned my addr 172.21.15.62:0/1134899996 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6259c640 1 --2- 172.21.15.62:0/1134899996 >> v2:172.21.15.145:3300/0 conn(0x7f0c640b44e0 0x7f0c640b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1373635811 0 0) 0x7f0c640ba170 con 0x7f0c64143bb0 2024-04-18T10:06:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c4c005740 con 0x7f0c64143bb0 2024-04-18T10:06:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 540984897 0 0) 0x7f0c4c005740 con 0x7f0c64143bb0 2024-04-18T10:06:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 >> v2:172.21.15.145:3300/0 conn(0x7f0c640b44e0 msgr2=0x7f0c640b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 --2- 172.21.15.62:0/1134899996 >> v2:172.21.15.145:3300/0 conn(0x7f0c640b44e0 0x7f0c640b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 >> v1:172.21.15.145:6789/0 conn(0x7f0c640b9d90 legacy=0x7f0c641450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c640b3fc0 con 0x7f0c64143bb0 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f0c5c001a00 con 0x7f0c64143bb0 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 >> v1:172.21.15.62:6789/0 conn(0x7f0c64143bb0 legacy=0x7f0c64144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 --2- 172.21.15.62:0/1134899996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c4c005bd0 0x7f0c4c005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 --2- 172.21.15.62:0/1134899996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c4c006a20 0x7f0c4c006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 --2- 172.21.15.62:0/1134899996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c4c0073e0 0x7f0c4c00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0c64145b80 con 0x7f0c4c0073e0 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f0c4c00e380 con 0x7f0c4c005bd0 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f0c4c00e580 con 0x7f0c4c006a20 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6359e640 1 --2- 172.21.15.62:0/1134899996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c4c0073e0 0x7f0c4c00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c62d9d640 1 --2- 172.21.15.62:0/1134899996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c4c005bd0 0x7f0c4c005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6259c640 1 --2- 172.21.15.62:0/1134899996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c4c006a20 0x7f0c4c006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6359e640 1 -- 172.21.15.62:0/1134899996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c4c006a20 msgr2=0x7f0c4c006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6359e640 1 --2- 172.21.15.62:0/1134899996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c4c006a20 0x7f0c4c006ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6359e640 1 -- 172.21.15.62:0/1134899996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c4c005bd0 msgr2=0x7f0c4c005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:11.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6359e640 1 --2- 172.21.15.62:0/1134899996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c4c005bd0 0x7f0c4c005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.414+0000 7f0c6359e640 1 -- 172.21.15.62:0/1134899996 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c4c005740 con 0x7f0c4c0073e0 2024-04-18T10:06:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6359e640 1 -- 172.21.15.62:0/1134899996 reap_dead start 2024-04-18T10:06:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6359e640 1 --2- 172.21.15.62:0/1134899996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c4c0073e0 0x7f0c4c00dbe0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00a840 tx=0x7f0c5c01fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26aeb3a7946f4fe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c00a460 con 0x7f0c4c0073e0 2024-04-18T10:06:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c5c0aa710 con 0x7f0c4c0073e0 2024-04-18T10:06:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c61d9b640 1 -- 172.21.15.62:0/1134899996 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c0aab00 con 0x7f0c4c0073e0 2024-04-18T10:06:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 -- 172.21.15.62:0/1134899996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c4c0073e0 msgr2=0x7f0c4c00dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 --2- 172.21.15.62:0/1134899996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c4c0073e0 0x7f0c4c00dbe0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00a840 tx=0x7f0c5c01fac0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.418 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 -- 172.21.15.62:0/1134899996 shutdown_connections 2024-04-18T10:06:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 --2- 172.21.15.62:0/1134899996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c4c0073e0 0x7f0c4c00dbe0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 -- 172.21.15.62:0/1134899996 >> 172.21.15.62:0/1134899996 conn(0x7f0c6406c0d0 msgr2=0x7f0c640b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 -- 172.21.15.62:0/1134899996 shutdown_connections 2024-04-18T10:06:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 -- 172.21.15.62:0/1134899996 wait complete. 2024-04-18T10:06:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 Processor -- start 2024-04-18T10:06:11.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 -- start start 2024-04-18T10:06:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c4c0073e0 0x7f0c640b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c641d8c20 0x7f0c640b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c6409c0e0 0x7f0c6409c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6409e3d0 con 0x7f0c6409c0e0 2024-04-18T10:06:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f0c640b4970 con 0x7f0c641d8c20 2024-04-18T10:06:11.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6900b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f0c640b17d0 con 0x7f0c4c0073e0 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6359e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c6409c0e0 0x7f0c6409c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6359e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c6409c0e0 0x7f0c6409c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41770/0 (socket says 172.21.15.62:41770) 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6359e640 1 -- 172.21.15.62:0/3394460197 learned_addr learned my addr 172.21.15.62:0/3394460197 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c62d9d640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c4c0073e0 0x7f0c640b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.418+0000 7f0c6259c640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c641d8c20 0x7f0c640b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c6359e640 1 -- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c4c0073e0 msgr2=0x7f0c640b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c6359e640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c4c0073e0 0x7f0c640b0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c6359e640 1 -- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c641d8c20 msgr2=0x7f0c640b1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c6359e640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c641d8c20 0x7f0c640b1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c6359e640 1 -- 172.21.15.62:0/3394460197 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c640b7cc0 con 0x7f0c6409c0e0 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c6359e640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c6409c0e0 0x7f0c6409c560 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c09c200 tx=0x7f0c4c007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c4b7fe640 1 -- 172.21.15.62:0/3394460197 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c0bb070 con 0x7f0c6409c0e0 2024-04-18T10:06:11.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c4b7fe640 1 -- 172.21.15.62:0/3394460197 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c5c0aa8c0 con 0x7f0c6409c0e0 2024-04-18T10:06:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c4b7fe640 1 -- 172.21.15.62:0/3394460197 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c00a460 con 0x7f0c6409c0e0 2024-04-18T10:06:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c6900b640 1 -- 172.21.15.62:0/3394460197 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c640b2cb0 con 0x7f0c6409c0e0 2024-04-18T10:06:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c6900b640 1 -- 172.21.15.62:0/3394460197 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c640b2f20 con 0x7f0c6409c0e0 2024-04-18T10:06:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c6900b640 1 -- 172.21.15.62:0/3394460197 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c640b1e10 con 0x7f0c6409c0e0 2024-04-18T10:06:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c4b7fe640 1 -- 172.21.15.62:0/3394460197 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0c5c0be070 con 0x7f0c6409c0e0 2024-04-18T10:06:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c4b7fe640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0c500683b0 0x7f0c5006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:11.423 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c4b7fe640 1 -- 172.21.15.62:0/3394460197 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0c5c123530 con 0x7f0c6409c0e0 2024-04-18T10:06:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c62d9d640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0c500683b0 0x7f0c5006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:11.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.422+0000 7f0c62d9d640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0c500683b0 0x7f0c5006a870 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0c54000c10 tx=0x7f0c54009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:11.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.426+0000 7f0c4b7fe640 1 -- 172.21.15.62:0/3394460197 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0c640b1e10 con 0x7f0c6409c0e0 2024-04-18T10:06:11.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 -- 172.21.15.62:0/3394460197 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0c64068b90 con 0x7f0c6409c0e0 2024-04-18T10:06:11.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c4b7fe640 1 -- 172.21.15.62:0/3394460197 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+73 (secure 0 0 0) 0x7f0c5c0197d0 con 0x7f0c6409c0e0 2024-04-18T10:06:11.625 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-18T10:06:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 -- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0c500683b0 msgr2=0x7f0c5006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0c500683b0 0x7f0c5006a870 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0c54000c10 tx=0x7f0c54009040 comp rx=0 tx=0).stop 2024-04-18T10:06:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 -- 172.21.15.62:0/3394460197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c6409c0e0 msgr2=0x7f0c6409c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c6409c0e0 0x7f0c6409c560 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c09c200 tx=0x7f0c4c007a50 comp rx=0 tx=0).stop 2024-04-18T10:06:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 -- 172.21.15.62:0/3394460197 shutdown_connections 2024-04-18T10:06:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0c500683b0 0x7f0c5006a870 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0c6409c0e0 0x7f0c6409c560 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0c641d8c20 0x7f0c640b1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 --2- 172.21.15.62:0/3394460197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0c4c0073e0 0x7f0c640b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:11.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 -- 172.21.15.62:0/3394460197 >> 172.21.15.62:0/3394460197 conn(0x7f0c6406c0d0 msgr2=0x7f0c64143390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 -- 172.21.15.62:0/3394460197 shutdown_connections 2024-04-18T10:06:11.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11.626+0000 7f0c6900b640 1 -- 172.21.15.62:0/3394460197 wait complete. 2024-04-18T10:06:11.643 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-18T10:06:11.643 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-18T10:06:11.643 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-04-18T10:06:11.643 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-18T10:06:11.643 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:11.643 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-18T10:06:11.643 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-18T10:06:11.643 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-18T10:06:11.643 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-18T10:06:11.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:11.644 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-04-18T10:06:11.649 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-04-18T10:06:11.650 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-18T10:06:11.651 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-04-18T10:06:11.651 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-18T10:06:11.651 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42503sec preferred_lft 42503sec 2024-04-18T10:06:11.651 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-04-18T10:06:11.651 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-18T10:06:11.651 INFO:teuthology.orchestra.run.smithi062.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-18T10:06:11.651 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 02:42:4b:60:3b:63 brd ff:ff:ff:ff:ff:ff 2024-04-18T10:06:11.651 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-18T10:06:11.651 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-18T10:06:11.651 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-18T10:06:11.652 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-18T10:06:11.652 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-04-18T10:06:11.652 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add name ceph-brx type bridge 2024-04-18T10:06:11.652 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip addr flush dev ceph-brx 2024-04-18T10:06:11.652 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set ceph-brx up 2024-04-18T10:06:11.652 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-18T10:06:11.652 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-04-18T10:06:11.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-18T10:06:11.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-18T10:06:11.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:11.778 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-18T10:06:11.828 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-04-18T10:06:11.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:11.829 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-04-18T10:06:11.898 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-04-18T10:06:11.898 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-18T10:06:11.898 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-18T10:06:11.899 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-18T10:06:11.899 INFO:teuthology.orchestra.run.smithi062.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-18T10:06:11.899 INFO:teuthology.orchestra.run.smithi062.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-18T10:06:11.899 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-18T10:06:11.899 INFO:teuthology.orchestra.run.smithi062.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-18T10:06:11.899 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-18T10:06:11.899 INFO:teuthology.orchestra.run.smithi062.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-18T10:06:11.899 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-18T10:06:11.900 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-18T10:06:11.900 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-04-18T10:06:11.900 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-18T10:06:11.900 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-18T10:06:11.900 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-18T10:06:11.900 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-04-18T10:06:11.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:11 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-18T10:06:12.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-18T10:06:12.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:12.024 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-04-18T10:06:12.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:12.031 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list-id 2024-04-18T10:06:12.075 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-18T10:06:12.076 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-04-18T10:06:12.076 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:12.076 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-18T10:06:12.076 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-04-18T10:06:12.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-18T10:06:12.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-18T10:06:12.196 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-18T10:06:12.197 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-18T10:06:12.197 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-04-18T10:06:12.197 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-18T10:06:12.197 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-18T10:06:12.197 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-18T10:06:12.197 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-18T10:06:12.197 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-18T10:06:12.197 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-04-18T10:06:12.244 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-18T10:06:12.298 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-18T10:06:12.305 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-18T10:06:12.305 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-04-18T10:06:12.305 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-04-18T10:06:12.305 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set dev brx.0 master ceph-brx 2024-04-18T10:06:12.305 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-04-18T10:06:12.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:12 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-18T10:06:12.403 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-18T10:06:12.410 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-18T10:06:12.410 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-18T10:06:12.410 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:12.455 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-18T10:06:12.455 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-18T10:06:12.456 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:12.503 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-04-18T10:06:12.557 DEBUG:teuthology.orchestra.run.smithi062:> cat /proc/self/mounts | awk '{print $2}' 2024-04-18T10:06:12.605 INFO:teuthology.orchestra.run.smithi062.stdout:/sys 2024-04-18T10:06:12.605 INFO:teuthology.orchestra.run.smithi062.stdout:/proc 2024-04-18T10:06:12.605 INFO:teuthology.orchestra.run.smithi062.stdout:/dev 2024-04-18T10:06:12.605 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/pts 2024-04-18T10:06:12.605 INFO:teuthology.orchestra.run.smithi062.stdout:/run 2024-04-18T10:06:12.605 INFO:teuthology.orchestra.run.smithi062.stdout:/ 2024-04-18T10:06:12.605 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/security 2024-04-18T10:06:12.605 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/shm 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/run/lock 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/cgroup 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/pstore 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/bpf 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/hugepages 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/mqueue 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/debug 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/tracing 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/fs/nfsd 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/fuse/connections 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/config 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-sysusers.service 2024-04-18T10:06:12.606 INFO:teuthology.orchestra.run.smithi062.stdout:/run/rpc_pipefs 2024-04-18T10:06:12.607 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/1000 2024-04-18T10:06:12.607 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph 2024-04-18T10:06:12.607 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-0 2024-04-18T10:06:12.607 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-1 2024-04-18T10:06:12.607 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-2 2024-04-18T10:06:12.607 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-3 2024-04-18T10:06:12.607 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns 2024-04-18T10:06:12.607 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:12.607 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:12.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:12.608 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-04-18T10:06:12.656 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-18T10:06:12.656 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-18T10:06:12.701 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-04-18T10:06:12.717 DEBUG:teuthology.orchestra.run.smithi062:> cat /proc/self/mounts | awk '{print $2}' 2024-04-18T10:06:12.764 INFO:teuthology.orchestra.run.smithi062.stdout:/sys 2024-04-18T10:06:12.764 INFO:teuthology.orchestra.run.smithi062.stdout:/proc 2024-04-18T10:06:12.764 INFO:teuthology.orchestra.run.smithi062.stdout:/dev 2024-04-18T10:06:12.764 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/pts 2024-04-18T10:06:12.764 INFO:teuthology.orchestra.run.smithi062.stdout:/run 2024-04-18T10:06:12.764 INFO:teuthology.orchestra.run.smithi062.stdout:/ 2024-04-18T10:06:12.764 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/security 2024-04-18T10:06:12.764 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/shm 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/run/lock 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/cgroup 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/pstore 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/bpf 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/hugepages 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/mqueue 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/debug 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/tracing 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/fs/nfsd 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/fuse/connections 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/config 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-sysusers.service 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/run/rpc_pipefs 2024-04-18T10:06:12.765 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/1000 2024-04-18T10:06:12.766 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph 2024-04-18T10:06:12.766 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-0 2024-04-18T10:06:12.766 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-1 2024-04-18T10:06:12.766 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-2 2024-04-18T10:06:12.766 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-3 2024-04-18T10:06:12.766 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns 2024-04-18T10:06:12.766 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:12.766 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:12.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:12.767 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-04-18T10:06:12.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-04-18T10:06:12.782+0000 7fb64ccd0580 -1 init, newargv = 0x555add158bc0 newargc=15 2024-04-18T10:06:12.781 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[34590]: starting ceph client 2024-04-18T10:06:12.787 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[34590]: starting fuse 2024-04-18T10:06:13.887 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-04-18T10:06:13.904 DEBUG:teuthology.orchestra.run.smithi062:> cat /proc/self/mounts | awk '{print $2}' 2024-04-18T10:06:13.953 INFO:teuthology.orchestra.run.smithi062.stdout:/sys 2024-04-18T10:06:13.953 INFO:teuthology.orchestra.run.smithi062.stdout:/proc 2024-04-18T10:06:13.953 INFO:teuthology.orchestra.run.smithi062.stdout:/dev 2024-04-18T10:06:13.953 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/pts 2024-04-18T10:06:13.953 INFO:teuthology.orchestra.run.smithi062.stdout:/run 2024-04-18T10:06:13.953 INFO:teuthology.orchestra.run.smithi062.stdout:/ 2024-04-18T10:06:13.953 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/security 2024-04-18T10:06:13.954 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/shm 2024-04-18T10:06:13.954 INFO:teuthology.orchestra.run.smithi062.stdout:/run/lock 2024-04-18T10:06:13.954 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/cgroup 2024-04-18T10:06:13.954 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/pstore 2024-04-18T10:06:13.954 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/bpf 2024-04-18T10:06:13.954 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-04-18T10:06:13.954 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/hugepages 2024-04-18T10:06:13.954 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/mqueue 2024-04-18T10:06:13.954 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/debug 2024-04-18T10:06:13.954 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/tracing 2024-04-18T10:06:13.954 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/fs/nfsd 2024-04-18T10:06:13.954 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/fuse/connections 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/config 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-sysusers.service 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/run/rpc_pipefs 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/1000 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-0 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-1 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-2 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-3 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:13.955 INFO:teuthology.orchestra.run.smithi062.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:13.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:13.956 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-04-18T10:06:14.004 INFO:teuthology.orchestra.run.smithi062.stdout:45 2024-04-18T10:06:14.004 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-18T10:06:14.004 DEBUG:teuthology.orchestra.run.smithi062:> sudo stdin-killer -- python3 -c ' 2024-04-18T10:06:14.004 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-04-18T10:06:14.004 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-04-18T10:06:14.004 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-04-18T10:06:14.004 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-04-18T10:06:14.005 DEBUG:teuthology.orchestra.run.smithi062:> 2024-04-18T10:06:14.005 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-04-18T10:06:14.005 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-18T10:06:14.005 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-04-18T10:06:14.005 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-18T10:06:14.005 DEBUG:teuthology.orchestra.run.smithi062:> 2024-04-18T10:06:14.005 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-04-18T10:06:14.005 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-04-18T10:06:14.005 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-04-18T10:06:14.005 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-04-18T10:06:14.005 DEBUG:teuthology.orchestra.run.smithi062:> 2024-04-18T10:06:14.005 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-04-18T10:06:14.006 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-18T10:06:14.006 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-18T10:06:14.006 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-18T10:06:14.006 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-04-18T10:06:14.006 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-04-18T10:06:14.006 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-04-18T10:06:14.006 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-18T10:06:14.006 DEBUG:teuthology.orchestra.run.smithi062:> 2024-04-18T10:06:14.006 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-04-18T10:06:14.006 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-04-18T10:06:14.132 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.34590.asok 2024-04-18T10:06:14.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-18T10:06:14.146 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.34590.asok 2024-04-18T10:06:14.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:14.146 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.34590.asok status 2024-04-18T10:06:14.298 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:06:14.298 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-04-18T10:06:14.298 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "a6549451253d26216d353246c0fe6f6d80f3b9cf", 2024-04-18T10:06:14.298 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 19.0.0-3115-ga6549451 (a6549451253d26216d353246c0fe6f6d80f3b9cf) squid (dev)", 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "34590", 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 4623, 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 4623 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-04-18T10:06:14.299 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1601433763 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 1601433763 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.4623 192.168.0.1:0/1601433763", 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/1601433763", 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 8, 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 21, 2024-04-18T10:06:14.300 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-04-18T10:06:14.301 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-04-18T10:06:14.301 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-04-18T10:06:14.301 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:06:14.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:14.313 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-04-18T10:06:14.313 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:14.320 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-04-18T10:06:14.320 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:14.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:14.321 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:14.379 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-18T10:06:14.394 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:14.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.550+0000 7f1da7573640 1 Processor -- start 2024-04-18T10:06:14.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da7573640 1 -- start start 2024-04-18T10:06:14.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da7573640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da00b4970 0x7f1da01442c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da7573640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1da00ba170 con 0x7f1da00b9d90 2024-04-18T10:06:14.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da7573640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1da01458a0 con 0x7f1da00b44e0 2024-04-18T10:06:14.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da7573640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f1da00b4de0 con 0x7f1da00b4970 2024-04-18T10:06:14.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da530f640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1da00b9d90 0x7f1da01450e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37072/0 (socket says 172.21.15.62:37072) 2024-04-18T10:06:14.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da530f640 1 -- 172.21.15.62:0/2651402181 learned_addr learned my addr 172.21.15.62:0/2651402181 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:14.557 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da5b10640 1 --2- 172.21.15.62:0/2651402181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da00b4970 0x7f1da01442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2254724714 0 0) 0x7f1da00ba170 con 0x7f1da00b9d90 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d8c003530 con 0x7f1da00b9d90 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4006609550 0 0) 0x7f1d8c003530 con 0x7f1da00b9d90 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da00b4970 msgr2=0x7f1da01442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 --2- 172.21.15.62:0/2651402181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da00b4970 0x7f1da01442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 >> v1:172.21.15.145:6789/0 conn(0x7f1da00b44e0 legacy=0x7f1da0143bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da00b3fc0 con 0x7f1da00b9d90 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f1d9c001680 con 0x7f1da00b9d90 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 >> v1:172.21.15.62:6789/0 conn(0x7f1da00b9d90 legacy=0x7f1da01450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 --2- 172.21.15.62:0/2651402181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da00b4970 0x7f1d8c0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 --2- 172.21.15.62:0/2651402181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d8c003f70 0x7f1d8c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 --2- 172.21.15.62:0/2651402181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d8c004d50 0x7f1d8c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1da00b4de0 con 0x7f1d8c004d50 2024-04-18T10:06:14.558 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f1d8c00bb60 con 0x7f1d8c003f70 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f1d8c00df70 con 0x7f1da00b4970 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da5b10640 1 --2- 172.21.15.62:0/2651402181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d8c004d50 0x7f1d8c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da530f640 1 --2- 172.21.15.62:0/2651402181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da00b4970 0x7f1d8c0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da4b0e640 1 --2- 172.21.15.62:0/2651402181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d8c003f70 0x7f1d8c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da5b10640 1 -- 172.21.15.62:0/2651402181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da00b4970 msgr2=0x7f1d8c0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da5b10640 1 --2- 172.21.15.62:0/2651402181 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da00b4970 0x7f1d8c0039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da5b10640 1 -- 172.21.15.62:0/2651402181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d8c003f70 msgr2=0x7f1d8c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da5b10640 1 --2- 172.21.15.62:0/2651402181 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d8c003f70 0x7f1d8c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.554+0000 7f1da5b10640 1 -- 172.21.15.62:0/2651402181 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d8c003530 con 0x7f1d8c004d50 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da5b10640 1 --2- 172.21.15.62:0/2651402181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d8c004d50 0x7f1d8c0051d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1d98009fc0 tx=0x7f1d9801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3796e4e3c8bf923a server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d980b2070 con 0x7f1d8c004d50 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d980a5b80 con 0x7f1d8c004d50 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1d97fff640 1 -- 172.21.15.62:0/2651402181 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98009c10 con 0x7f1d8c004d50 2024-04-18T10:06:14.559 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 -- 172.21.15.62:0/2651402181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d8c004d50 msgr2=0x7f1d8c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:14.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 --2- 172.21.15.62:0/2651402181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d8c004d50 0x7f1d8c0051d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1d98009fc0 tx=0x7f1d9801cc80 comp rx=0 tx=0).stop 2024-04-18T10:06:14.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da5b10640 1 -- 172.21.15.62:0/2651402181 reap_dead start 2024-04-18T10:06:14.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 -- 172.21.15.62:0/2651402181 shutdown_connections 2024-04-18T10:06:14.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 -- 172.21.15.62:0/2651402181 >> 172.21.15.62:0/2651402181 conn(0x7f1da006c0d0 msgr2=0x7f1da00b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:14.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 -- 172.21.15.62:0/2651402181 shutdown_connections 2024-04-18T10:06:14.560 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 -- 172.21.15.62:0/2651402181 wait complete. 2024-04-18T10:06:14.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 Processor -- start 2024-04-18T10:06:14.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 -- start start 2024-04-18T10:06:14.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da009c4b0 0x7f1da01e09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1da01e0ef0 0x7f1da013dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1da013e130 0x7f1da01dd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.562 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1da009e3d0 con 0x7f1da013e130 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f1da0145b80 con 0x7f1da01e0ef0 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da7573640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f1da01e1580 con 0x7f1da009c4b0 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.558+0000 7f1da5b10640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1da013e130 0x7f1da01dd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da5b10640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1da013e130 0x7f1da01dd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41784/0 (socket says 172.21.15.62:41784) 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da5b10640 1 -- 172.21.15.62:0/4147779334 learned_addr learned my addr 172.21.15.62:0/4147779334 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da4b0e640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1da01e0ef0 0x7f1da013dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da530f640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da009c4b0 0x7f1da01e09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da5b10640 1 -- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da009c4b0 msgr2=0x7f1da01e09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da5b10640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da009c4b0 0x7f1da01e09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da5b10640 1 -- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1da01e0ef0 msgr2=0x7f1da013dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da5b10640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1da01e0ef0 0x7f1da013dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da5b10640 1 -- 172.21.15.62:0/4147779334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da013e5b0 con 0x7f1da013e130 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da5b10640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1da013e130 0x7f1da01dd440 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1d980b0040 tx=0x7f1d8c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1d95ffb640 1 -- 172.21.15.62:0/4147779334 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d980be070 con 0x7f1da013e130 2024-04-18T10:06:14.563 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1d95ffb640 1 -- 172.21.15.62:0/4147779334 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d980b3040 con 0x7f1da013e130 2024-04-18T10:06:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da7573640 1 -- 172.21.15.62:0/4147779334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da0070880 con 0x7f1da013e130 2024-04-18T10:06:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1d95ffb640 1 -- 172.21.15.62:0/4147779334 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d980b2050 con 0x7f1da013e130 2024-04-18T10:06:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da7573640 1 -- 172.21.15.62:0/4147779334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da0143730 con 0x7f1da013e130 2024-04-18T10:06:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da7573640 1 -- 172.21.15.62:0/4147779334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da0068b90 con 0x7f1da013e130 2024-04-18T10:06:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1d95ffb640 1 -- 172.21.15.62:0/4147779334 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1d980c1070 con 0x7f1da013e130 2024-04-18T10:06:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1d95ffb640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1d840683b0 0x7f1d8406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1d95ffb640 1 -- 172.21.15.62:0/4147779334 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1d98120eb0 con 0x7f1da013e130 2024-04-18T10:06:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da530f640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1d840683b0 0x7f1d8406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.562+0000 7f1da530f640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1d840683b0 0x7f1d8406a870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c004850 tx=0x7f1d9c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:14.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.566+0000 7f1d95ffb640 1 -- 172.21.15.62:0/4147779334 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1da013e5b0 con 0x7f1da013e130 2024-04-18T10:06:14.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.730+0000 7f1da7573640 1 -- 172.21.15.62:0/4147779334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1da01e1700 con 0x7f1da013e130 2024-04-18T10:06:14.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1d95ffb640 1 -- 172.21.15.62:0/4147779334 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f1d98016990 con 0x7f1da013e130 2024-04-18T10:06:14.733 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:14.733 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:14.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1da7573640 1 -- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1d840683b0 msgr2=0x7f1d8406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:14.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1da7573640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1d840683b0 0x7f1d8406a870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c004850 tx=0x7f1d9c009040 comp rx=0 tx=0).stop 2024-04-18T10:06:14.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1da7573640 1 -- 172.21.15.62:0/4147779334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1da013e130 msgr2=0x7f1da01dd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:14.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1da7573640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1da013e130 0x7f1da01dd440 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1d980b0040 tx=0x7f1d8c005840 comp rx=0 tx=0).stop 2024-04-18T10:06:14.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1da7573640 1 -- 172.21.15.62:0/4147779334 shutdown_connections 2024-04-18T10:06:14.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1da7573640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1d840683b0 0x7f1d8406a870 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1da7573640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1da013e130 0x7f1da01dd440 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1da7573640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1da01e0ef0 0x7f1da013dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1da7573640 1 --2- 172.21.15.62:0/4147779334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1da009c4b0 0x7f1da01e09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1da7573640 1 -- 172.21.15.62:0/4147779334 >> 172.21.15.62:0/4147779334 conn(0x7f1da006c0d0 msgr2=0x7f1da0074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:14.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1da7573640 1 -- 172.21.15.62:0/4147779334 shutdown_connections 2024-04-18T10:06:14.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.734+0000 7f1da7573640 1 -- 172.21.15.62:0/4147779334 wait complete. 2024-04-18T10:06:14.752 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-18T10:05:56.782649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-18T10:05:57.541623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:14.752 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:14.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea6d6e1640 1 Processor -- start 2024-04-18T10:06:14.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea6d6e1640 1 -- start start 2024-04-18T10:06:14.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea6d6e1640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fea68146690 0x7fea6809e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea6d6e1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea6806d4f0 con 0x7fea6806e490 2024-04-18T10:06:14.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea6d6e1640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea6806d7d0 con 0x7fea680b4c90 2024-04-18T10:06:14.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea6d6e1640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fea6806d2a0 con 0x7fea68146690 2024-04-18T10:06:14.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea677fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fea6806e490 0x7fea68150410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37082/0 (socket says 172.21.15.62:37082) 2024-04-18T10:06:14.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea677fe640 1 -- 172.21.15.62:0/933228212 learned_addr learned my addr 172.21.15.62:0/933228212 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:14.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea667fc640 1 --2- 172.21.15.62:0/933228212 >> v2:172.21.15.145:3300/0 conn(0x7fea68146690 0x7fea6809e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2645846538 0 0) 0x7fea6806d4f0 con 0x7fea6806e490 2024-04-18T10:06:14.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea50003530 con 0x7fea6806e490 2024-04-18T10:06:14.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 886722475 0 0) 0x7fea50003530 con 0x7fea6806e490 2024-04-18T10:06:14.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 >> v2:172.21.15.145:3300/0 conn(0x7fea68146690 msgr2=0x7fea6809e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:14.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 --2- 172.21.15.62:0/933228212 >> v2:172.21.15.145:3300/0 conn(0x7fea68146690 0x7fea6809e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 >> v1:172.21.15.145:6789/0 conn(0x7fea680b4c90 legacy=0x7fea68145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:14.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6809c570 con 0x7fea6806e490 2024-04-18T10:06:14.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fea60001a00 con 0x7fea6806e490 2024-04-18T10:06:14.880 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 >> v1:172.21.15.62:6789/0 conn(0x7fea6806e490 legacy=0x7fea68150410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 --2- 172.21.15.62:0/933228212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea500039c0 0x7fea50003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 --2- 172.21.15.62:0/933228212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea50004810 0x7fea50004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 --2- 172.21.15.62:0/933228212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea500051d0 0x7fea5000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fea6806d2a0 con 0x7fea500039c0 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fea5000c170 con 0x7fea50004810 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fea5000c370 con 0x7fea500051d0 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.878+0000 7fea66ffd640 1 --2- 172.21.15.62:0/933228212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea500039c0 0x7fea50003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea677fe640 1 --2- 172.21.15.62:0/933228212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea500051d0 0x7fea5000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea667fc640 1 --2- 172.21.15.62:0/933228212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea50004810 0x7fea50004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea66ffd640 1 -- 172.21.15.62:0/933228212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea500051d0 msgr2=0x7fea5000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea66ffd640 1 --2- 172.21.15.62:0/933228212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea500051d0 0x7fea5000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea66ffd640 1 -- 172.21.15.62:0/933228212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea50004810 msgr2=0x7fea50004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea66ffd640 1 --2- 172.21.15.62:0/933228212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea50004810 0x7fea50004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea66ffd640 1 -- 172.21.15.62:0/933228212 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea50003530 con 0x7fea500039c0 2024-04-18T10:06:14.881 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea677fe640 1 -- 172.21.15.62:0/933228212 reap_dead start 2024-04-18T10:06:14.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea66ffd640 1 --2- 172.21.15.62:0/933228212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea500039c0 0x7fea50003de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fea58001780 tx=0x7fea5801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4780590c58ac3168 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:14.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea580013a0 con 0x7fea500039c0 2024-04-18T10:06:14.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea580a4d40 con 0x7fea500039c0 2024-04-18T10:06:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea65ffb640 1 -- 172.21.15.62:0/933228212 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea58005310 con 0x7fea500039c0 2024-04-18T10:06:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea6d6e1640 1 -- 172.21.15.62:0/933228212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea500039c0 msgr2=0x7fea50003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea6d6e1640 1 --2- 172.21.15.62:0/933228212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea500039c0 0x7fea50003de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fea58001780 tx=0x7fea5801c400 comp rx=0 tx=0).stop 2024-04-18T10:06:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea6d6e1640 1 -- 172.21.15.62:0/933228212 shutdown_connections 2024-04-18T10:06:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea6d6e1640 1 --2- 172.21.15.62:0/933228212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea500039c0 0x7fea50003de0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea6d6e1640 1 -- 172.21.15.62:0/933228212 >> 172.21.15.62:0/933228212 conn(0x7fea680b9690 msgr2=0x7fea680b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea6d6e1640 1 -- 172.21.15.62:0/933228212 shutdown_connections 2024-04-18T10:06:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.882+0000 7fea6d6e1640 1 -- 172.21.15.62:0/933228212 wait complete. 2024-04-18T10:06:14.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea6d6e1640 1 Processor -- start 2024-04-18T10:06:14.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea6d6e1640 1 -- start start 2024-04-18T10:06:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea6d6e1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea681e1e90 0x7fea681e2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea6d6e1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea680704e0 0x7fea681431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea6d6e1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea680710f0 0x7fea68071570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea6d6e1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fea68149220 con 0x7fea681e1e90 2024-04-18T10:06:14.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea6d6e1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fea681490a0 con 0x7fea680704e0 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea6d6e1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fea681493a0 con 0x7fea680710f0 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea66ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea681e1e90 0x7fea681e2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea66ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea681e1e90 0x7fea681e2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41798/0 (socket says 172.21.15.62:41798) 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea66ffd640 1 -- 172.21.15.62:0/2205522576 learned_addr learned my addr 172.21.15.62:0/2205522576 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea667fc640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea680704e0 0x7fea681431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea677fe640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea680710f0 0x7fea68071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea66ffd640 1 -- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea680710f0 msgr2=0x7fea68071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea66ffd640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea680710f0 0x7fea68071570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea66ffd640 1 -- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea680704e0 msgr2=0x7fea681431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea66ffd640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea680704e0 0x7fea681431d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea66ffd640 1 -- 172.21.15.62:0/2205522576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea68071ab0 con 0x7fea681e1e90 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea66ffd640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea681e1e90 0x7fea681e2270 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fea58005040 tx=0x7fea50005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:14.886 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea4f7fe640 1 -- 172.21.15.62:0/2205522576 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea580ba070 con 0x7fea681e1e90 2024-04-18T10:06:14.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea4f7fe640 1 -- 172.21.15.62:0/2205522576 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea580a4b20 con 0x7fea681e1e90 2024-04-18T10:06:14.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea4f7fe640 1 -- 172.21.15.62:0/2205522576 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea58007a40 con 0x7fea681e1e90 2024-04-18T10:06:14.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea6d6e1640 1 -- 172.21.15.62:0/2205522576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea681e2890 con 0x7fea681e1e90 2024-04-18T10:06:14.887 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea6d6e1640 1 -- 172.21.15.62:0/2205522576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea680728d0 con 0x7fea681e1e90 2024-04-18T10:06:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea6d6e1640 1 -- 172.21.15.62:0/2205522576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea68068c00 con 0x7fea681e1e90 2024-04-18T10:06:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.886+0000 7fea4f7fe640 1 -- 172.21.15.62:0/2205522576 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fea580bd070 con 0x7fea681e1e90 2024-04-18T10:06:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.890+0000 7fea4f7fe640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fea540683b0 0x7fea5406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.890+0000 7fea4f7fe640 1 -- 172.21.15.62:0/2205522576 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fea5811f880 con 0x7fea681e1e90 2024-04-18T10:06:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.890+0000 7fea667fc640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fea540683b0 0x7fea5406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:14.888 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.890+0000 7fea667fc640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fea540683b0 0x7fea5406a870 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea680b3720 tx=0x7fea5c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:14.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:14.894+0000 7fea4f7fe640 1 -- 172.21.15.62:0/2205522576 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fea68071ab0 con 0x7fea681e1e90 2024-04-18T10:06:15.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 -- 172.21.15.62:0/2205522576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fea680b44c0 con 0x7fea681e1e90 2024-04-18T10:06:15.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea4f7fe640 1 -- 172.21.15.62:0/2205522576 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fea58016110 con 0x7fea681e1e90 2024-04-18T10:06:15.052 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:15.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 -- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fea540683b0 msgr2=0x7fea5406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fea540683b0 0x7fea5406a870 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fea680b3720 tx=0x7fea5c008040 comp rx=0 tx=0).stop 2024-04-18T10:06:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 -- 172.21.15.62:0/2205522576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea681e1e90 msgr2=0x7fea681e2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea681e1e90 0x7fea681e2270 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fea58005040 tx=0x7fea50005840 comp rx=0 tx=0).stop 2024-04-18T10:06:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 -- 172.21.15.62:0/2205522576 shutdown_connections 2024-04-18T10:06:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fea540683b0 0x7fea5406a870 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea680710f0 0x7fea68071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea680704e0 0x7fea681431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 --2- 172.21.15.62:0/2205522576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea681e1e90 0x7fea681e2270 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 -- 172.21.15.62:0/2205522576 >> 172.21.15.62:0/2205522576 conn(0x7fea680b9690 msgr2=0x7fea6809d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 -- 172.21.15.62:0/2205522576 shutdown_connections 2024-04-18T10:06:15.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.054+0000 7fea6d6e1640 1 -- 172.21.15.62:0/2205522576 wait complete. 2024-04-18T10:06:15.069 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-18T10:06:15.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1596a12640 1 Processor -- start 2024-04-18T10:06:15.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1596a12640 1 -- start start 2024-04-18T10:06:15.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1596a12640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f159013fe50 0x7f15900b5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1596a12640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15900b4250 0x7f1590150320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1596a12640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15900b4800 con 0x7f15900b7cf0 2024-04-18T10:06:15.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1596a12640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f15900b5090 con 0x7f159013fe50 2024-04-18T10:06:15.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1596a12640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f1590140230 con 0x7f15900b4250 2024-04-18T10:06:15.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158f7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f15900b7cf0 0x7f15900bbd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37084/0 (socket says 172.21.15.62:37084) 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158f7fe640 1 -- 172.21.15.62:0/4239813764 learned_addr learned my addr 172.21.15.62:0/4239813764 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158ffff640 1 --2- 172.21.15.62:0/4239813764 >> v2:172.21.15.145:3300/0 conn(0x7f159013fe50 0x7f15900b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1594faf640 1 --2- 172.21.15.62:0/4239813764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15900b4250 0x7f1590150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2426391840 0 0) 0x7f15900b4800 con 0x7f15900b7cf0 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1578003530 con 0x7f15900b7cf0 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1672704379 0 0) 0x7f1578003530 con 0x7f15900b7cf0 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15900b4250 msgr2=0x7f1590150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 --2- 172.21.15.62:0/4239813764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15900b4250 0x7f1590150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 >> v2:172.21.15.145:3300/0 conn(0x7f159013fe50 msgr2=0x7f15900b5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 --2- 172.21.15.62:0/4239813764 >> v2:172.21.15.145:3300/0 conn(0x7f159013fe50 0x7f15900b5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15900b4ae0 con 0x7f15900b7cf0 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f1584001680 con 0x7f15900b7cf0 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 >> v1:172.21.15.62:6789/0 conn(0x7f15900b7cf0 legacy=0x7f15900bbd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 --2- 172.21.15.62:0/4239813764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15900b4250 0x7f15780039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 --2- 172.21.15.62:0/4239813764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1578003f40 0x7f1578004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 --2- 172.21.15.62:0/4239813764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1578004d50 0x7f15780051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f15900b5090 con 0x7f1578004d50 2024-04-18T10:06:15.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f1590140230 con 0x7f1578003f40 2024-04-18T10:06:15.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f157800b9d0 con 0x7f15900b4250 2024-04-18T10:06:15.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1594faf640 1 --2- 172.21.15.62:0/4239813764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1578004d50 0x7f15780051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1594faf640 1 -- 172.21.15.62:0/4239813764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15900b4250 msgr2=0x7f15780039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:15.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1594faf640 1 --2- 172.21.15.62:0/4239813764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15900b4250 0x7f15780039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1594faf640 1 -- 172.21.15.62:0/4239813764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1578003f40 msgr2=0x7f1578004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:15.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1594faf640 1 --2- 172.21.15.62:0/4239813764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1578003f40 0x7f1578004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.198+0000 7f1594faf640 1 -- 172.21.15.62:0/4239813764 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1578003530 con 0x7f1578004d50 2024-04-18T10:06:15.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1594faf640 1 --2- 172.21.15.62:0/4239813764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1578004d50 0x7f15780051d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1588009ae0 tx=0x7f158801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fbda954da7b2c6f server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:15.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15880b2070 con 0x7f1578004d50 2024-04-18T10:06:15.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15880a56a0 con 0x7f1578004d50 2024-04-18T10:06:15.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f158effd640 1 -- 172.21.15.62:0/4239813764 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15880096b0 con 0x7f1578004d50 2024-04-18T10:06:15.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 -- 172.21.15.62:0/4239813764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1578004d50 msgr2=0x7f15780051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 --2- 172.21.15.62:0/4239813764 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1578004d50 0x7f15780051d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1588009ae0 tx=0x7f158801ca50 comp rx=0 tx=0).stop 2024-04-18T10:06:15.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1594faf640 1 -- 172.21.15.62:0/4239813764 reap_dead start 2024-04-18T10:06:15.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 -- 172.21.15.62:0/4239813764 shutdown_connections 2024-04-18T10:06:15.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 -- 172.21.15.62:0/4239813764 >> 172.21.15.62:0/4239813764 conn(0x7f1590067ea0 msgr2=0x7f15900bb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:15.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 -- 172.21.15.62:0/4239813764 shutdown_connections 2024-04-18T10:06:15.202 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 -- 172.21.15.62:0/4239813764 wait complete. 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 Processor -- start 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 -- start start 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1590146190 0x7f1590139b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1590144620 0x7f15901dab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1590138470 0x7f15901388e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f15900b2cc0 con 0x7f1590138470 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f15900b2b40 con 0x7f1590144620 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1596a12640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f15900b6380 con 0x7f1590146190 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1594faf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1590138470 0x7f15901388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1594faf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1590138470 0x7f15901388e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41824/0 (socket says 172.21.15.62:41824) 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f1594faf640 1 -- 172.21.15.62:0/4042633739 learned_addr learned my addr 172.21.15.62:0/4042633739 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.202+0000 7f158ffff640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1590146190 0x7f1590139b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.204 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f158f7fe640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1590144620 0x7f15901dab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f1594faf640 1 -- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1590146190 msgr2=0x7f1590139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f1594faf640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1590146190 0x7f1590139b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f1594faf640 1 -- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1590144620 msgr2=0x7f15901dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f1594faf640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1590144620 0x7f15901dab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f1594faf640 1 -- 172.21.15.62:0/4042633739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1590146830 con 0x7f1590138470 2024-04-18T10:06:15.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f1594faf640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1590138470 0x7f15901388e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f15880b0040 tx=0x7f1578005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:15.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f158cff9640 1 -- 172.21.15.62:0/4042633739 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15880bb070 con 0x7f1590138470 2024-04-18T10:06:15.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f158cff9640 1 -- 172.21.15.62:0/4042633739 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1588000e50 con 0x7f1590138470 2024-04-18T10:06:15.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f1596a12640 1 -- 172.21.15.62:0/4042633739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1590146570 con 0x7f1590138470 2024-04-18T10:06:15.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f158cff9640 1 -- 172.21.15.62:0/4042633739 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15880b2040 con 0x7f1590138470 2024-04-18T10:06:15.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f1596a12640 1 -- 172.21.15.62:0/4042633739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15901441a0 con 0x7f1590138470 2024-04-18T10:06:15.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f1596a12640 1 -- 172.21.15.62:0/4042633739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1590069230 con 0x7f1590138470 2024-04-18T10:06:15.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f158cff9640 1 -- 172.21.15.62:0/4042633739 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f15880be070 con 0x7f1590138470 2024-04-18T10:06:15.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f158cff9640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f157c0683b0 0x7f157c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f158cff9640 1 -- 172.21.15.62:0/4042633739 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f158811faa0 con 0x7f1590138470 2024-04-18T10:06:15.206 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f158ffff640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f157c0683b0 0x7f157c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.206+0000 7f158ffff640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f157c0683b0 0x7f157c06a870 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1580004b40 tx=0x7f158000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:15.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.210+0000 7f158cff9640 1 -- 172.21.15.62:0/4042633739 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f15880167c0 con 0x7f1590138470 2024-04-18T10:06:15.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 -- 172.21.15.62:0/4042633739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1590146b60 con 0x7f1590138470 2024-04-18T10:06:15.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f158cff9640 1 -- 172.21.15.62:0/4042633739 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f15880efa10 con 0x7f1590138470 2024-04-18T10:06:15.372 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-18T10:06:15.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 -- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f157c0683b0 msgr2=0x7f157c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f157c0683b0 0x7f157c06a870 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1580004b40 tx=0x7f158000b040 comp rx=0 tx=0).stop 2024-04-18T10:06:15.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 -- 172.21.15.62:0/4042633739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1590138470 msgr2=0x7f15901388e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1590138470 0x7f15901388e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f15880b0040 tx=0x7f1578005840 comp rx=0 tx=0).stop 2024-04-18T10:06:15.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 -- 172.21.15.62:0/4042633739 shutdown_connections 2024-04-18T10:06:15.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f157c0683b0 0x7f157c06a870 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1590138470 0x7f15901388e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1590144620 0x7f15901dab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 --2- 172.21.15.62:0/4042633739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1590146190 0x7f1590139b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 -- 172.21.15.62:0/4042633739 >> 172.21.15.62:0/4042633739 conn(0x7f1590067ea0 msgr2=0x7f15900b8860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:15.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 -- 172.21.15.62:0/4042633739 shutdown_connections 2024-04-18T10:06:15.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.374+0000 7f1596a12640 1 -- 172.21.15.62:0/4042633739 wait complete. 2024-04-18T10:06:15.389 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-18T10:06:15.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d62f4f640 1 Processor -- start 2024-04-18T10:06:15.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d62f4f640 1 -- start start 2024-04-18T10:06:15.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d62f4f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d5c0b01a0 0x7f7d5c06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d62f4f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7d5c0ae4e0 0x7f7d5c13e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d62f4f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d5c0b0580 con 0x7f7d5c0737c0 2024-04-18T10:06:15.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d62f4f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f7d5c0b59d0 con 0x7f7d5c0ae4e0 2024-04-18T10:06:15.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d62f4f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7d5c0ae970 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d60ceb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d5c0b01a0 0x7f7d5c06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d5bfff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7d5c0ae4e0 0x7f7d5c13e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d60ceb640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d5c0b01a0 0x7f7d5c06d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:52580/0 (socket says 172.21.15.62:52580) 2024-04-18T10:06:15.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d60ceb640 1 -- 172.21.15.62:0/3526415641 learned_addr learned my addr 172.21.15.62:0/3526415641 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:15.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d5bfff640 1 -- 172.21.15.62:0/3526415641 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d5c0b01a0 msgr2=0x7f7d5c06d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:15.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d5bfff640 1 --2- 172.21.15.62:0/3526415641 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d5c0b01a0 0x7f7d5c06d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.506 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d5bfff640 1 -- 172.21.15.62:0/3526415641 >> v1:172.21.15.145:6789/0 conn(0x7f7d5c0737c0 legacy=0x7f7d5c148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d5bfff640 1 -- 172.21.15.62:0/3526415641 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5c0b53a0 con 0x7f7d5c0ae4e0 2024-04-18T10:06:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.506+0000 7f7d60ceb640 1 --2- 172.21.15.62:0/3526415641 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d5c0b01a0 0x7f7d5c06d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:06:15.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5bfff640 1 --2- 172.21.15.62:0/3526415641 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d5c0ae4e0 0x7f7d5c13e980 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7d480097c0 tx=0x7f7d4801c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40e2a348bb12959c server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:15.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5b7fe640 1 -- 172.21.15.62:0/3526415641 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d480b0070 con 0x7f7d5c0ae4e0 2024-04-18T10:06:15.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5b7fe640 1 -- 172.21.15.62:0/3526415641 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d480ab600 con 0x7f7d5c0ae4e0 2024-04-18T10:06:15.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5b7fe640 1 -- 172.21.15.62:0/3526415641 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d480ab9d0 con 0x7f7d5c0ae4e0 2024-04-18T10:06:15.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 -- 172.21.15.62:0/3526415641 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d5c0ae4e0 msgr2=0x7f7d5c13e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 --2- 172.21.15.62:0/3526415641 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d5c0ae4e0 0x7f7d5c13e980 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7d480097c0 tx=0x7f7d4801c380 comp rx=0 tx=0).stop 2024-04-18T10:06:15.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 -- 172.21.15.62:0/3526415641 shutdown_connections 2024-04-18T10:06:15.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 --2- 172.21.15.62:0/3526415641 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d5c0ae4e0 0x7f7d5c13e980 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 --2- 172.21.15.62:0/3526415641 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d5c0b01a0 0x7f7d5c06d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 -- 172.21.15.62:0/3526415641 >> 172.21.15.62:0/3526415641 conn(0x7f7d5c0b9690 msgr2=0x7f7d5c0b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:15.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 -- 172.21.15.62:0/3526415641 shutdown_connections 2024-04-18T10:06:15.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 -- 172.21.15.62:0/3526415641 wait complete. 2024-04-18T10:06:15.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 Processor -- start 2024-04-18T10:06:15.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 -- start start 2024-04-18T10:06:15.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d5c0737c0 0x7f7d5c09d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d5c0b01a0 0x7f7d5c13f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d5c13fa40 0x7f7d5c136640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5c06e3e0 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5c06e260 con 0x7f7d5c0737c0 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d62f4f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7d5c09aa50 con 0x7f7d5c13fa40 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d5c0b01a0 0x7f7d5c13f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d5c0b01a0 0x7f7d5c13f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41832/0 (socket says 172.21.15.62:41832) 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5bfff640 1 -- 172.21.15.62:0/643710550 learned_addr learned my addr 172.21.15.62:0/643710550 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d60ceb640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d5c0737c0 0x7f7d5c09d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d614ec640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d5c13fa40 0x7f7d5c136640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5bfff640 1 -- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d5c13fa40 msgr2=0x7f7d5c136640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5bfff640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d5c13fa40 0x7f7d5c136640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5bfff640 1 -- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d5c0737c0 msgr2=0x7f7d5c09d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5bfff640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d5c0737c0 0x7f7d5c09d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5bfff640 1 -- 172.21.15.62:0/643710550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5c1411f0 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.513 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.510+0000 7f7d5bfff640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d5c0b01a0 0x7f7d5c13f500 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7d48019fb0 tx=0x7f7d48019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:15.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.514+0000 7f7d597fa640 1 -- 172.21.15.62:0/643710550 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d480b8020 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.514+0000 7f7d597fa640 1 -- 172.21.15.62:0/643710550 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d480b4410 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.514+0000 7f7d62f4f640 1 -- 172.21.15.62:0/643710550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d5c0732d0 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.514+0000 7f7d597fa640 1 -- 172.21.15.62:0/643710550 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d480b0040 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.514+0000 7f7d62f4f640 1 -- 172.21.15.62:0/643710550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d5c141680 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.514+0000 7f7d62f4f640 1 -- 172.21.15.62:0/643710550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d5c068c00 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.514+0000 7f7d597fa640 1 -- 172.21.15.62:0/643710550 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7d480bb020 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.514+0000 7f7d597fa640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7d400683b0 0x7f7d4006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.514+0000 7f7d597fa640 1 -- 172.21.15.62:0/643710550 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7d48120840 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.514+0000 7f7d60ceb640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7d400683b0 0x7f7d4006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.518+0000 7f7d60ceb640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7d400683b0 0x7f7d4006a870 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000bce0 tx=0x7f7d5000b450 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:15.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.518+0000 7f7d597fa640 1 -- 172.21.15.62:0/643710550 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7d480c2030 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.678+0000 7f7d62f4f640 1 -- 172.21.15.62:0/643710550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7d5c140f10 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.678+0000 7f7d597fa640 1 -- 172.21.15.62:0/643710550 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f7d48016090 con 0x7f7d5c0b01a0 2024-04-18T10:06:15.677 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-04-18T10:06:15.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.682+0000 7f7d62f4f640 1 -- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7d400683b0 msgr2=0x7f7d4006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.682+0000 7f7d62f4f640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7d400683b0 0x7f7d4006a870 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7d5000bce0 tx=0x7f7d5000b450 comp rx=0 tx=0).stop 2024-04-18T10:06:15.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.682+0000 7f7d62f4f640 1 -- 172.21.15.62:0/643710550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d5c0b01a0 msgr2=0x7f7d5c13f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.682+0000 7f7d62f4f640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d5c0b01a0 0x7f7d5c13f500 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7d48019fb0 tx=0x7f7d48019120 comp rx=0 tx=0).stop 2024-04-18T10:06:15.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.682+0000 7f7d62f4f640 1 -- 172.21.15.62:0/643710550 shutdown_connections 2024-04-18T10:06:15.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.682+0000 7f7d62f4f640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7d400683b0 0x7f7d4006a870 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.682+0000 7f7d62f4f640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d5c13fa40 0x7f7d5c136640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.682+0000 7f7d62f4f640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d5c0b01a0 0x7f7d5c13f500 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.682+0000 7f7d62f4f640 1 --2- 172.21.15.62:0/643710550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d5c0737c0 0x7f7d5c09d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.682+0000 7f7d62f4f640 1 -- 172.21.15.62:0/643710550 >> 172.21.15.62:0/643710550 conn(0x7f7d5c0b9690 msgr2=0x7f7d5c0b4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:15.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.682+0000 7f7d62f4f640 1 -- 172.21.15.62:0/643710550 shutdown_connections 2024-04-18T10:06:15.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.682+0000 7f7d62f4f640 1 -- 172.21.15.62:0/643710550 wait complete. 2024-04-18T10:06:15.697 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:15.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.822+0000 7fdb99863640 1 Processor -- start 2024-04-18T10:06:15.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb99863640 1 -- start start 2024-04-18T10:06:15.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb99863640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdb940b3e20 0x7fdb940afae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb99863640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb94144390 con 0x7fdb9409cc40 2024-04-18T10:06:15.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb99863640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb9409c4d0 con 0x7fdb9406d470 2024-04-18T10:06:15.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb99863640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fdb940b5b40 con 0x7fdb940b3e20 2024-04-18T10:06:15.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb937fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fdb9409cc40 0x7fdb9414ea20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37098/0 (socket says 172.21.15.62:37098) 2024-04-18T10:06:15.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb937fe640 1 -- 172.21.15.62:0/740896148 learned_addr learned my addr 172.21.15.62:0/740896148 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:15.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb927fc640 1 --2- 172.21.15.62:0/740896148 >> v2:172.21.15.145:3300/0 conn(0x7fdb940b3e20 0x7fdb940afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3755287268 0 0) 0x7fdb94144390 con 0x7fdb9409cc40 2024-04-18T10:06:15.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb80003530 con 0x7fdb9409cc40 2024-04-18T10:06:15.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 113424564 0 0) 0x7fdb80003530 con 0x7fdb9409cc40 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 >> v2:172.21.15.145:3300/0 conn(0x7fdb940b3e20 msgr2=0x7fdb940afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 --2- 172.21.15.62:0/740896148 >> v2:172.21.15.145:3300/0 conn(0x7fdb940b3e20 0x7fdb940afae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 >> v1:172.21.15.145:6789/0 conn(0x7fdb9406d470 legacy=0x7fdb94144b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb9406d850 con 0x7fdb9409cc40 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fdb8c001680 con 0x7fdb9409cc40 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 >> v1:172.21.15.62:6789/0 conn(0x7fdb9409cc40 legacy=0x7fdb9414ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 --2- 172.21.15.62:0/740896148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb800039c0 0x7fdb80003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 --2- 172.21.15.62:0/740896148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb80004810 0x7fdb80004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 --2- 172.21.15.62:0/740896148 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb800052f0 0x7fdb8000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8000c190 con 0x7fdb80004810 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fdb8000c340 con 0x7fdb800039c0 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fdb8000c540 con 0x7fdb800052f0 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb927fc640 1 --2- 172.21.15.62:0/740896148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb80004810 0x7fdb80004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb92ffd640 1 --2- 172.21.15.62:0/740896148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb800039c0 0x7fdb80003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb927fc640 1 -- 172.21.15.62:0/740896148 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb800052f0 msgr2=0x7fdb8000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:15.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb927fc640 1 --2- 172.21.15.62:0/740896148 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb800052f0 0x7fdb8000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb927fc640 1 -- 172.21.15.62:0/740896148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb800039c0 msgr2=0x7fdb80003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:15.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb927fc640 1 --2- 172.21.15.62:0/740896148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb800039c0 0x7fdb80003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb927fc640 1 -- 172.21.15.62:0/740896148 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb80003530 con 0x7fdb80004810 2024-04-18T10:06:15.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.826+0000 7fdb937fe640 1 -- 172.21.15.62:0/740896148 reap_dead start 2024-04-18T10:06:15.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb927fc640 1 --2- 172.21.15.62:0/740896148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb80004810 0x7fdb80004c90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00aab0 tx=0x7fdb7c01df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743add41b83e7a8c server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:15.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c0b3070 con 0x7fdb80004810 2024-04-18T10:06:15.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb7c0a6ce0 con 0x7fdb80004810 2024-04-18T10:06:15.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb91ffb640 1 -- 172.21.15.62:0/740896148 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c00b040 con 0x7fdb80004810 2024-04-18T10:06:15.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 -- 172.21.15.62:0/740896148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb80004810 msgr2=0x7fdb80004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 --2- 172.21.15.62:0/740896148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb80004810 0x7fdb80004c90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c00aab0 tx=0x7fdb7c01df80 comp rx=0 tx=0).stop 2024-04-18T10:06:15.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 -- 172.21.15.62:0/740896148 shutdown_connections 2024-04-18T10:06:15.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 --2- 172.21.15.62:0/740896148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb80004810 0x7fdb80004c90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 -- 172.21.15.62:0/740896148 >> 172.21.15.62:0/740896148 conn(0x7fdb9406c0d0 msgr2=0x7fdb940b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:15.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 -- 172.21.15.62:0/740896148 shutdown_connections 2024-04-18T10:06:15.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 -- 172.21.15.62:0/740896148 wait complete. 2024-04-18T10:06:15.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 Processor -- start 2024-04-18T10:06:15.830 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 -- start start 2024-04-18T10:06:15.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb80004810 0x7fdb9413f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb94071ba0 0x7fdb941e4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb940725e0 0x7fdb94147960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdb940afeb0 con 0x7fdb940725e0 2024-04-18T10:06:15.832 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fdb940afd30 con 0x7fdb80004810 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb99863640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fdb940b0030 con 0x7fdb94071ba0 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb937fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb940725e0 0x7fdb94147960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb937fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb940725e0 0x7fdb94147960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41844/0 (socket says 172.21.15.62:41844) 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb937fe640 1 -- 172.21.15.62:0/2338539023 learned_addr learned my addr 172.21.15.62:0/2338539023 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb927fc640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb94071ba0 0x7fdb941e4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb92ffd640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb80004810 0x7fdb9413f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb937fe640 1 -- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb94071ba0 msgr2=0x7fdb941e4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb937fe640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb94071ba0 0x7fdb941e4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb937fe640 1 -- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb80004810 msgr2=0x7fdb9413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb937fe640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb80004810 0x7fdb9413f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb937fe640 1 -- 172.21.15.62:0/2338539023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb941d17e0 con 0x7fdb940725e0 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.830+0000 7fdb937fe640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb940725e0 0x7fdb94147960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0011f0 tx=0x7fdb8c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:15.833 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.834+0000 7fdb8bfff640 1 -- 172.21.15.62:0/2338539023 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c007230 con 0x7fdb940725e0 2024-04-18T10:06:15.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.834+0000 7fdb8bfff640 1 -- 172.21.15.62:0/2338539023 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb8c0073f0 con 0x7fdb940725e0 2024-04-18T10:06:15.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.834+0000 7fdb99863640 1 -- 172.21.15.62:0/2338539023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb94072d60 con 0x7fdb940725e0 2024-04-18T10:06:15.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.834+0000 7fdb8bfff640 1 -- 172.21.15.62:0/2338539023 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c00ce50 con 0x7fdb940725e0 2024-04-18T10:06:15.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.834+0000 7fdb99863640 1 -- 172.21.15.62:0/2338539023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb94071680 con 0x7fdb940725e0 2024-04-18T10:06:15.834 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.834+0000 7fdb99863640 1 -- 172.21.15.62:0/2338539023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb941d0b30 con 0x7fdb940725e0 2024-04-18T10:06:15.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.834+0000 7fdb8bfff640 1 -- 172.21.15.62:0/2338539023 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdb8c011070 con 0x7fdb940725e0 2024-04-18T10:06:15.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.834+0000 7fdb8bfff640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdb74068320 0x7fdb7406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:15.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.834+0000 7fdb8bfff640 1 -- 172.21.15.62:0/2338539023 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdb8c085fa0 con 0x7fdb940725e0 2024-04-18T10:06:15.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.834+0000 7fdb92ffd640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdb74068320 0x7fdb7406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:15.835 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.834+0000 7fdb92ffd640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdb74068320 0x7fdb7406a7e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdb84001d20 tx=0x7fdb84009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:15.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:15.838+0000 7fdb8bfff640 1 -- 172.21.15.62:0/2338539023 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdb941d0b30 con 0x7fdb940725e0 2024-04-18T10:06:16.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.006+0000 7fdb99863640 1 -- 172.21.15.62:0/2338539023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb94068b90 con 0x7fdb940725e0 2024-04-18T10:06:16.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.006+0000 7fdb8bfff640 1 -- 172.21.15.62:0/2338539023 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7fdb8c01a060 con 0x7fdb940725e0 2024-04-18T10:06:16.006 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:16.006 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:16.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.010+0000 7fdb99863640 1 -- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdb74068320 msgr2=0x7fdb7406a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.010+0000 7fdb99863640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdb74068320 0x7fdb7406a7e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdb84001d20 tx=0x7fdb84009040 comp rx=0 tx=0).stop 2024-04-18T10:06:16.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.010+0000 7fdb99863640 1 -- 172.21.15.62:0/2338539023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb940725e0 msgr2=0x7fdb94147960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.010+0000 7fdb99863640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb940725e0 0x7fdb94147960 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0011f0 tx=0x7fdb8c00cb10 comp rx=0 tx=0).stop 2024-04-18T10:06:16.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.010+0000 7fdb99863640 1 -- 172.21.15.62:0/2338539023 shutdown_connections 2024-04-18T10:06:16.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.010+0000 7fdb99863640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdb74068320 0x7fdb7406a7e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.010+0000 7fdb99863640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb940725e0 0x7fdb94147960 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.010+0000 7fdb99863640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb94071ba0 0x7fdb941e4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.010+0000 7fdb99863640 1 --2- 172.21.15.62:0/2338539023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb80004810 0x7fdb9413f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.010+0000 7fdb99863640 1 -- 172.21.15.62:0/2338539023 >> 172.21.15.62:0/2338539023 conn(0x7fdb9406c0d0 msgr2=0x7fdb940aedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:16.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.010+0000 7fdb99863640 1 -- 172.21.15.62:0/2338539023 shutdown_connections 2024-04-18T10:06:16.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.010+0000 7fdb99863640 1 -- 172.21.15.62:0/2338539023 wait complete. 2024-04-18T10:06:16.025 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-18T10:05:56.782649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-18T10:05:57.541623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:16.025 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:16.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce7b64640 1 Processor -- start 2024-04-18T10:06:16.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce7b64640 1 -- start start 2024-04-18T10:06:16.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce7b64640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbce0067a70 0x7fbce0072cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce7b64640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbce0067e50 con 0x7fbce0070ce0 2024-04-18T10:06:16.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce7b64640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbce0073420 con 0x7fbce0149510 2024-04-18T10:06:16.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce7b64640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fbce0073200 con 0x7fbce0067a70 2024-04-18T10:06:16.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce50ff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbce0070ce0 0x7fbce013fbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37104/0 (socket says 172.21.15.62:37104) 2024-04-18T10:06:16.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce50ff640 1 -- 172.21.15.62:0/4142494317 learned_addr learned my addr 172.21.15.62:0/4142494317 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:16.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce5900640 1 --2- 172.21.15.62:0/4142494317 >> v2:172.21.15.145:3300/0 conn(0x7fbce0067a70 0x7fbce0072cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3759556951 0 0) 0x7fbce0067e50 con 0x7fbce0070ce0 2024-04-18T10:06:16.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbccc003530 con 0x7fbce0070ce0 2024-04-18T10:06:16.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1145468020 0 0) 0x7fbccc003530 con 0x7fbce0070ce0 2024-04-18T10:06:16.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 >> v2:172.21.15.145:3300/0 conn(0x7fbce0067a70 msgr2=0x7fbce0072cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:16.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 --2- 172.21.15.62:0/4142494317 >> v2:172.21.15.145:3300/0 conn(0x7fbce0067a70 0x7fbce0072cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 >> v1:172.21.15.145:6789/0 conn(0x7fbce0149510 legacy=0x7fbce0149980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce0073700 con 0x7fbce0070ce0 2024-04-18T10:06:16.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fbcd0001680 con 0x7fbce0070ce0 2024-04-18T10:06:16.155 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 >> v1:172.21.15.62:6789/0 conn(0x7fbce0070ce0 legacy=0x7fbce013fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 --2- 172.21.15.62:0/4142494317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbccc0039c0 0x7fbccc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 --2- 172.21.15.62:0/4142494317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbccc004810 0x7fbccc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 --2- 172.21.15.62:0/4142494317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbccc0052f0 0x7fbccc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbce0073200 con 0x7fbccc004810 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fbccc00c1c0 con 0x7fbccc0052f0 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fbccc00c3c0 con 0x7fbccc0039c0 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce50ff640 1 --2- 172.21.15.62:0/4142494317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbccc004810 0x7fbccc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce5900640 1 --2- 172.21.15.62:0/4142494317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbccc0039c0 0x7fbccc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce6101640 1 --2- 172.21.15.62:0/4142494317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbccc0052f0 0x7fbccc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce50ff640 1 -- 172.21.15.62:0/4142494317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbccc0039c0 msgr2=0x7fbccc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce50ff640 1 --2- 172.21.15.62:0/4142494317 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbccc0039c0 0x7fbccc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce50ff640 1 -- 172.21.15.62:0/4142494317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbccc0052f0 msgr2=0x7fbccc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce50ff640 1 --2- 172.21.15.62:0/4142494317 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbccc0052f0 0x7fbccc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.156 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce50ff640 1 -- 172.21.15.62:0/4142494317 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbccc003530 con 0x7fbccc004810 2024-04-18T10:06:16.157 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.154+0000 7fbce6101640 1 -- 172.21.15.62:0/4142494317 reap_dead start 2024-04-18T10:06:16.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce50ff640 1 --2- 172.21.15.62:0/4142494317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbccc004810 0x7fbccc004c90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000c210 tx=0x7fbcd001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473f819417ab8076 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:16.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0007d20 con 0x7fbccc004810 2024-04-18T10:06:16.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd00a9070 con 0x7fbccc004810 2024-04-18T10:06:16.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce48fe640 1 -- 172.21.15.62:0/4142494317 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0005da0 con 0x7fbccc004810 2024-04-18T10:06:16.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 -- 172.21.15.62:0/4142494317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbccc004810 msgr2=0x7fbccc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 --2- 172.21.15.62:0/4142494317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbccc004810 0x7fbccc004c90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000c210 tx=0x7fbcd001cc60 comp rx=0 tx=0).stop 2024-04-18T10:06:16.158 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 -- 172.21.15.62:0/4142494317 shutdown_connections 2024-04-18T10:06:16.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 --2- 172.21.15.62:0/4142494317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbccc004810 0x7fbccc004c90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 -- 172.21.15.62:0/4142494317 >> 172.21.15.62:0/4142494317 conn(0x7fbce00b9cf0 msgr2=0x7fbce00ba100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:16.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 -- 172.21.15.62:0/4142494317 shutdown_connections 2024-04-18T10:06:16.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 -- 172.21.15.62:0/4142494317 wait complete. 2024-04-18T10:06:16.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 Processor -- start 2024-04-18T10:06:16.159 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 -- start start 2024-04-18T10:06:16.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbccc004810 0x7fbce01e1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbce01413c0 0x7fbce0148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbce013c2a0 0x7fbce0136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbce006be60 con 0x7fbce01413c0 2024-04-18T10:06:16.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fbce006bce0 con 0x7fbce013c2a0 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce7b64640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fbce006bfe0 con 0x7fbccc004810 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.158+0000 7fbce50ff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbce01413c0 0x7fbce0148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce50ff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbce01413c0 0x7fbce0148130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41864/0 (socket says 172.21.15.62:41864) 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce50ff640 1 -- 172.21.15.62:0/1763219679 learned_addr learned my addr 172.21.15.62:0/1763219679 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce6101640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbce013c2a0 0x7fbce0136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce5900640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbccc004810 0x7fbce01e1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce50ff640 1 -- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbccc004810 msgr2=0x7fbce01e1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce50ff640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbccc004810 0x7fbce01e1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce50ff640 1 -- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbce013c2a0 msgr2=0x7fbce0136de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce50ff640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbce013c2a0 0x7fbce0136de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce50ff640 1 -- 172.21.15.62:0/1763219679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce01418a0 con 0x7fbce01413c0 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce6101640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbce013c2a0 0x7fbce0136de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:16.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce50ff640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbce01413c0 0x7fbce0148130 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000c210 tx=0x7fbccc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:16.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbcd67fc640 1 -- 172.21.15.62:0/1763219679 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0007d20 con 0x7fbce01413c0 2024-04-18T10:06:16.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbcd67fc640 1 -- 172.21.15.62:0/1763219679 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd0000ee0 con 0x7fbce01413c0 2024-04-18T10:06:16.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbcd67fc640 1 -- 172.21.15.62:0/1763219679 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0001680 con 0x7fbce01413c0 2024-04-18T10:06:16.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce7b64640 1 -- 172.21.15.62:0/1763219679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce0148df0 con 0x7fbce01413c0 2024-04-18T10:06:16.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce7b64640 1 -- 172.21.15.62:0/1763219679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce013bd80 con 0x7fbce01413c0 2024-04-18T10:06:16.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce7b64640 1 -- 172.21.15.62:0/1763219679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbce013bb70 con 0x7fbce01413c0 2024-04-18T10:06:16.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbcd67fc640 1 -- 172.21.15.62:0/1763219679 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbcd0007180 con 0x7fbce01413c0 2024-04-18T10:06:16.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbcd67fc640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbcc40683b0 0x7fbcc406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbcd67fc640 1 -- 172.21.15.62:0/1763219679 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbcd0120130 con 0x7fbce01413c0 2024-04-18T10:06:16.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.162+0000 7fbce5900640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbcc40683b0 0x7fbcc406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.166+0000 7fbce5900640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbcc40683b0 0x7fbcc406a870 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009ab0 tx=0x7fbcdc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:16.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.166+0000 7fbcd67fc640 1 -- 172.21.15.62:0/1763219679 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbcd0016970 con 0x7fbce01413c0 2024-04-18T10:06:16.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.334+0000 7fbce7b64640 1 -- 172.21.15.62:0/1763219679 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbce0068c00 con 0x7fbce01413c0 2024-04-18T10:06:16.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.334+0000 7fbcd67fc640 1 -- 172.21.15.62:0/1763219679 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fbcd00f0780 con 0x7fbce01413c0 2024-04-18T10:06:16.333 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:16.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.334+0000 7fbce7b64640 1 -- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbcc40683b0 msgr2=0x7fbcc406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.334+0000 7fbce7b64640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbcc40683b0 0x7fbcc406a870 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc009ab0 tx=0x7fbcdc008040 comp rx=0 tx=0).stop 2024-04-18T10:06:16.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.334+0000 7fbce7b64640 1 -- 172.21.15.62:0/1763219679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbce01413c0 msgr2=0x7fbce0148130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.334+0000 7fbce7b64640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbce01413c0 0x7fbce0148130 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbcd000c210 tx=0x7fbccc005840 comp rx=0 tx=0).stop 2024-04-18T10:06:16.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.338+0000 7fbce7b64640 1 -- 172.21.15.62:0/1763219679 shutdown_connections 2024-04-18T10:06:16.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.338+0000 7fbce7b64640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbcc40683b0 0x7fbcc406a870 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.338+0000 7fbce7b64640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbce013c2a0 0x7fbce0136de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.338+0000 7fbce7b64640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbce01413c0 0x7fbce0148130 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.338+0000 7fbce7b64640 1 --2- 172.21.15.62:0/1763219679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbccc004810 0x7fbce01e1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.338+0000 7fbce7b64640 1 -- 172.21.15.62:0/1763219679 >> 172.21.15.62:0/1763219679 conn(0x7fbce00b9cf0 msgr2=0x7fbce013ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:16.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.338+0000 7fbce7b64640 1 -- 172.21.15.62:0/1763219679 shutdown_connections 2024-04-18T10:06:16.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.338+0000 7fbce7b64640 1 -- 172.21.15.62:0/1763219679 wait complete. 2024-04-18T10:06:16.352 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-18T10:06:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.482+0000 7fe841904640 1 Processor -- start 2024-04-18T10:06:16.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.482+0000 7fe841904640 1 -- start start 2024-04-18T10:06:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe841904640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe83c06dd90 0x7fe83c0b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe841904640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83c147840 0x7fe83c147c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe841904640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe83c148650 con 0x7fe83c1481c0 2024-04-18T10:06:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe841904640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fe83c0af620 con 0x7fe83c06dd90 2024-04-18T10:06:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe841904640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fe83c06e170 con 0x7fe83c147840 2024-04-18T10:06:16.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe83affd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe83c06dd90 0x7fe83c0b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe83b7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe83c1481c0 0x7fe83c143940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37120/0 (socket says 172.21.15.62:37120) 2024-04-18T10:06:16.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe83b7fe640 1 -- 172.21.15.62:0/3925469167 learned_addr learned my addr 172.21.15.62:0/3925469167 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:16.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe83a7fc640 1 --2- 172.21.15.62:0/3925469167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83c147840 0x7fe83c147c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe839ffb640 1 -- 172.21.15.62:0/3925469167 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 217438727 0 0) 0x7fe83c148650 con 0x7fe83c1481c0 2024-04-18T10:06:16.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe839ffb640 1 -- 172.21.15.62:0/3925469167 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe824004640 con 0x7fe83c1481c0 2024-04-18T10:06:16.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe83affd640 1 -- 172.21.15.62:0/3925469167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83c147840 msgr2=0x7fe83c147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:16.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe83affd640 1 --2- 172.21.15.62:0/3925469167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83c147840 0x7fe83c147c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe83affd640 1 -- 172.21.15.62:0/3925469167 >> v1:172.21.15.62:6789/0 conn(0x7fe83c1481c0 legacy=0x7fe83c143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe83affd640 1 -- 172.21.15.62:0/3925469167 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83c0af850 con 0x7fe83c06dd90 2024-04-18T10:06:16.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe83affd640 1 --2- 172.21.15.62:0/3925469167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe83c06dd90 0x7fe83c0b43a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0097c0 tx=0x7fe82c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ee4aabb11b1716 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe839ffb640 1 -- 172.21.15.62:0/3925469167 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c0b0070 con 0x7fe83c06dd90 2024-04-18T10:06:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe839ffb640 1 -- 172.21.15.62:0/3925469167 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe82c0ab400 con 0x7fe83c06dd90 2024-04-18T10:06:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.486+0000 7fe839ffb640 1 -- 172.21.15.62:0/3925469167 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c0ab7b0 con 0x7fe83c06dd90 2024-04-18T10:06:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 -- 172.21.15.62:0/3925469167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe83c06dd90 msgr2=0x7fe83c0b43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 --2- 172.21.15.62:0/3925469167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe83c06dd90 0x7fe83c0b43a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0097c0 tx=0x7fe82c01c410 comp rx=0 tx=0).stop 2024-04-18T10:06:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 -- 172.21.15.62:0/3925469167 shutdown_connections 2024-04-18T10:06:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 --2- 172.21.15.62:0/3925469167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83c147840 0x7fe83c147c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 --2- 172.21.15.62:0/3925469167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe83c06dd90 0x7fe83c0b43a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 -- 172.21.15.62:0/3925469167 >> 172.21.15.62:0/3925469167 conn(0x7fe83c0b9690 msgr2=0x7fe83c0b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 -- 172.21.15.62:0/3925469167 shutdown_connections 2024-04-18T10:06:16.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 -- 172.21.15.62:0/3925469167 wait complete. 2024-04-18T10:06:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 Processor -- start 2024-04-18T10:06:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 -- start start 2024-04-18T10:06:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe83c06dd90 0x7fe83c0b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83c147840 0x7fe83c1de330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe83c1481c0 0x7fe83c1de870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c0b5e20 con 0x7fe83c1481c0 2024-04-18T10:06:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c0b5ca0 con 0x7fe83c06dd90 2024-04-18T10:06:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fe83c144050 con 0x7fe83c147840 2024-04-18T10:06:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe83b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe83c1481c0 0x7fe83c1de870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe83b7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe83c1481c0 0x7fe83c1de870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41876/0 (socket says 172.21.15.62:41876) 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe83b7fe640 1 -- 172.21.15.62:0/1253496241 learned_addr learned my addr 172.21.15.62:0/1253496241 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe83a7fc640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83c147840 0x7fe83c1de330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe83b7fe640 1 -- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83c147840 msgr2=0x7fe83c1de330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe83b7fe640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83c147840 0x7fe83c1de330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe83affd640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe83c06dd90 0x7fe83c0b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe83b7fe640 1 -- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe83c06dd90 msgr2=0x7fe83c0b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe83b7fe640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe83c06dd90 0x7fe83c0b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe83b7fe640 1 -- 172.21.15.62:0/1253496241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83c1491b0 con 0x7fe83c1481c0 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe83affd640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe83c06dd90 0x7fe83c0b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe83b7fe640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe83c1481c0 0x7fe83c1de870 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe83400c3c0 tx=0x7fe83400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe823fff640 1 -- 172.21.15.62:0/1253496241 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834017070 con 0x7fe83c1481c0 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe823fff640 1 -- 172.21.15.62:0/1253496241 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe834005cf0 con 0x7fe83c1481c0 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe823fff640 1 -- 172.21.15.62:0/1253496241 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834007310 con 0x7fe83c1481c0 2024-04-18T10:06:16.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 -- 172.21.15.62:0/1253496241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe83c1423b0 con 0x7fe83c1481c0 2024-04-18T10:06:16.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 -- 172.21.15.62:0/1253496241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe83c072ff0 con 0x7fe83c1481c0 2024-04-18T10:06:16.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.490+0000 7fe841904640 1 -- 172.21.15.62:0/1253496241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe83c0731c0 con 0x7fe83c1481c0 2024-04-18T10:06:16.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.494+0000 7fe823fff640 1 -- 172.21.15.62:0/1253496241 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe83401a070 con 0x7fe83c1481c0 2024-04-18T10:06:16.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.494+0000 7fe823fff640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe828068380 0x7fe82806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.494+0000 7fe823fff640 1 -- 172.21.15.62:0/1253496241 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe834085c60 con 0x7fe83c1481c0 2024-04-18T10:06:16.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.494+0000 7fe83affd640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe828068380 0x7fe82806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.494+0000 7fe83affd640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe828068380 0x7fe82806a840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0ad000 tx=0x7fe82c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:16.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.498+0000 7fe823fff640 1 -- 172.21.15.62:0/1253496241 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe834056210 con 0x7fe83c1481c0 2024-04-18T10:06:16.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.662+0000 7fe841904640 1 -- 172.21.15.62:0/1253496241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe83c147fa0 con 0x7fe83c1481c0 2024-04-18T10:06:16.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.662+0000 7fe823fff640 1 -- 172.21.15.62:0/1253496241 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fe834055b10 con 0x7fe83c1481c0 2024-04-18T10:06:16.660 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-18T10:06:16.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.662+0000 7fe841904640 1 -- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe828068380 msgr2=0x7fe82806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.662+0000 7fe841904640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe828068380 0x7fe82806a840 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe82c0ad000 tx=0x7fe82c008040 comp rx=0 tx=0).stop 2024-04-18T10:06:16.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.662+0000 7fe841904640 1 -- 172.21.15.62:0/1253496241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe83c1481c0 msgr2=0x7fe83c1de870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.662+0000 7fe841904640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe83c1481c0 0x7fe83c1de870 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe83400c3c0 tx=0x7fe83400c880 comp rx=0 tx=0).stop 2024-04-18T10:06:16.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.666+0000 7fe841904640 1 -- 172.21.15.62:0/1253496241 shutdown_connections 2024-04-18T10:06:16.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.666+0000 7fe841904640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe828068380 0x7fe82806a840 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.666+0000 7fe841904640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe83c1481c0 0x7fe83c1de870 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.666+0000 7fe841904640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe83c147840 0x7fe83c1de330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.666+0000 7fe841904640 1 --2- 172.21.15.62:0/1253496241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe83c06dd90 0x7fe83c0b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.666+0000 7fe841904640 1 -- 172.21.15.62:0/1253496241 >> 172.21.15.62:0/1253496241 conn(0x7fe83c0b9690 msgr2=0x7fe83c09b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:16.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.666+0000 7fe841904640 1 -- 172.21.15.62:0/1253496241 shutdown_connections 2024-04-18T10:06:16.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.666+0000 7fe841904640 1 -- 172.21.15.62:0/1253496241 wait complete. 2024-04-18T10:06:16.682 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-18T10:06:16.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30cb6b5640 1 Processor -- start 2024-04-18T10:06:16.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30cb6b5640 1 -- start start 2024-04-18T10:06:16.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30cb6b5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f30c4146690 0x7f30c409e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30cb6b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c406e490 0x7f30c409e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30cb6b5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30c406d4f0 con 0x7f30c40b4c90 2024-04-18T10:06:16.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30cb6b5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f30c406d2a0 con 0x7f30c4146690 2024-04-18T10:06:16.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30cb6b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f30c406d7d0 con 0x7f30c406e490 2024-04-18T10:06:16.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30c9451640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f30c40b4c90 0x7f30c4145f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37124/0 (socket says 172.21.15.62:37124) 2024-04-18T10:06:16.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30c9451640 1 -- 172.21.15.62:0/1073998113 learned_addr learned my addr 172.21.15.62:0/1073998113 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:16.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30c8c50640 1 --2- 172.21.15.62:0/1073998113 >> v2:172.21.15.145:3300/0 conn(0x7f30c4146690 0x7f30c409e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30c9c52640 1 --2- 172.21.15.62:0/1073998113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c406e490 0x7f30c409e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890273074 0 0) 0x7f30c406d4f0 con 0x7f30c40b4c90 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30b0003530 con 0x7f30c40b4c90 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4109999668 0 0) 0x7f30b0003530 con 0x7f30c40b4c90 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c406e490 msgr2=0x7f30c409e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 --2- 172.21.15.62:0/1073998113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c406e490 0x7f30c409e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 >> v2:172.21.15.145:3300/0 conn(0x7f30c4146690 msgr2=0x7f30c409e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 --2- 172.21.15.62:0/1073998113 >> v2:172.21.15.145:3300/0 conn(0x7f30c4146690 0x7f30c409e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c409c570 con 0x7f30c40b4c90 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f30c0001680 con 0x7f30c40b4c90 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 >> v1:172.21.15.62:6789/0 conn(0x7f30c40b4c90 legacy=0x7f30c4145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 --2- 172.21.15.62:0/1073998113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30b00039c0 0x7f30b0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 --2- 172.21.15.62:0/1073998113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c406e490 0x7f30b0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.814+0000 7f30bbfff640 1 --2- 172.21.15.62:0/1073998113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0004d50 0x7f30b00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f30c406d2a0 con 0x7f30b00039c0 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f30c406d7d0 con 0x7f30b0004d50 2024-04-18T10:06:16.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f30b000b9d0 con 0x7f30c406e490 2024-04-18T10:06:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30c9451640 1 --2- 172.21.15.62:0/1073998113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30b00039c0 0x7f30b0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30c8c50640 1 --2- 172.21.15.62:0/1073998113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c406e490 0x7f30b0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30c9c52640 1 --2- 172.21.15.62:0/1073998113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0004d50 0x7f30b00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30c9451640 1 -- 172.21.15.62:0/1073998113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c406e490 msgr2=0x7f30b0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30c9451640 1 --2- 172.21.15.62:0/1073998113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30c406e490 0x7f30b0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30c9451640 1 -- 172.21.15.62:0/1073998113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0004d50 msgr2=0x7f30b00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30c9451640 1 --2- 172.21.15.62:0/1073998113 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0004d50 0x7f30b00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30c9451640 1 -- 172.21.15.62:0/1073998113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b0003530 con 0x7f30b00039c0 2024-04-18T10:06:16.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30c9451640 1 --2- 172.21.15.62:0/1073998113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30b00039c0 0x7f30b0003de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f30c0001060 tx=0x7f30c001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e60c7e30e43f58c server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:16.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c0007d80 con 0x7f30b00039c0 2024-04-18T10:06:16.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c00a9020 con 0x7f30b00039c0 2024-04-18T10:06:16.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.818+0000 7f30bbfff640 1 -- 172.21.15.62:0/1073998113 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c0005b00 con 0x7f30b00039c0 2024-04-18T10:06:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 -- 172.21.15.62:0/1073998113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30b00039c0 msgr2=0x7f30b0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 --2- 172.21.15.62:0/1073998113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30b00039c0 0x7f30b0003de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f30c0001060 tx=0x7f30c001cca0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30c9c52640 1 -- 172.21.15.62:0/1073998113 reap_dead start 2024-04-18T10:06:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 -- 172.21.15.62:0/1073998113 shutdown_connections 2024-04-18T10:06:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 -- 172.21.15.62:0/1073998113 >> 172.21.15.62:0/1073998113 conn(0x7f30c40b9690 msgr2=0x7f30c40b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 -- 172.21.15.62:0/1073998113 shutdown_connections 2024-04-18T10:06:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 -- 172.21.15.62:0/1073998113 wait complete. 2024-04-18T10:06:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 Processor -- start 2024-04-18T10:06:16.820 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 -- start start 2024-04-18T10:06:16.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00039c0 0x7f30c4072aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c4072fe0 0x7f30c41e1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30c41e1eb0 0x7f30c4141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f30c4150da0 con 0x7f30c4072fe0 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f30c4150c20 con 0x7f30c41e1eb0 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f30c4150f20 con 0x7f30b00039c0 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30c8c50640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c4072fe0 0x7f30c41e1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30c8c50640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c4072fe0 0x7f30c41e1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41900/0 (socket says 172.21.15.62:41900) 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30c8c50640 1 -- 172.21.15.62:0/3552301927 learned_addr learned my addr 172.21.15.62:0/3552301927 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30c9451640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00039c0 0x7f30c4072aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30c9c52640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30c41e1eb0 0x7f30c4141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30c8c50640 1 -- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00039c0 msgr2=0x7f30c4072aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30c8c50640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00039c0 0x7f30c4072aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30c8c50640 1 -- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30c41e1eb0 msgr2=0x7f30c4141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30c8c50640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30c41e1eb0 0x7f30c4141190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30c8c50640 1 -- 172.21.15.62:0/3552301927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30c413e8b0 con 0x7f30c4072fe0 2024-04-18T10:06:16.823 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30c8c50640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c4072fe0 0x7f30c41e1970 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f30b40119d0 tx=0x7f30b4011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:16.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30b9ffb640 1 -- 172.21.15.62:0/3552301927 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b400fb50 con 0x7f30c4072fe0 2024-04-18T10:06:16.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30b9ffb640 1 -- 172.21.15.62:0/3552301927 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30b400fcf0 con 0x7f30c4072fe0 2024-04-18T10:06:16.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 -- 172.21.15.62:0/3552301927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30c4142dc0 con 0x7f30c4072fe0 2024-04-18T10:06:16.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30b9ffb640 1 -- 172.21.15.62:0/3552301927 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30b400fea0 con 0x7f30c4072fe0 2024-04-18T10:06:16.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 -- 172.21.15.62:0/3552301927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30c4142a70 con 0x7f30c4072fe0 2024-04-18T10:06:16.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.822+0000 7f30cb6b5640 1 -- 172.21.15.62:0/3552301927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30c4068c00 con 0x7f30c4072fe0 2024-04-18T10:06:16.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.826+0000 7f30b9ffb640 1 -- 172.21.15.62:0/3552301927 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f30b4012040 con 0x7f30c4072fe0 2024-04-18T10:06:16.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.826+0000 7f30b9ffb640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f30a80683b0 0x7f30a806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:16.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.826+0000 7f30b9ffb640 1 -- 172.21.15.62:0/3552301927 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f30b4086f00 con 0x7f30c4072fe0 2024-04-18T10:06:16.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.826+0000 7f30c9451640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f30a80683b0 0x7f30a806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:16.825 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.826+0000 7f30c9451640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f30a80683b0 0x7f30a806a870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f30c00aa000 tx=0x7f30c0001a50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:16.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.830+0000 7f30b9ffb640 1 -- 172.21.15.62:0/3552301927 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f30b4019030 con 0x7f30c4072fe0 2024-04-18T10:06:16.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.990+0000 7f30cb6b5640 1 -- 172.21.15.62:0/3552301927 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f30c4144f40 con 0x7f30c4072fe0 2024-04-18T10:06:16.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30b9ffb640 1 -- 172.21.15.62:0/3552301927 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f30b4056fc0 con 0x7f30c4072fe0 2024-04-18T10:06:16.994 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-04-18T10:06:16.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30cb6b5640 1 -- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f30a80683b0 msgr2=0x7f30a806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30cb6b5640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f30a80683b0 0x7f30a806a870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f30c00aa000 tx=0x7f30c0001a50 comp rx=0 tx=0).stop 2024-04-18T10:06:16.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30cb6b5640 1 -- 172.21.15.62:0/3552301927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c4072fe0 msgr2=0x7f30c41e1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:16.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30cb6b5640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c4072fe0 0x7f30c41e1970 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f30b40119d0 tx=0x7f30b4011e90 comp rx=0 tx=0).stop 2024-04-18T10:06:16.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30cb6b5640 1 -- 172.21.15.62:0/3552301927 shutdown_connections 2024-04-18T10:06:16.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30cb6b5640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f30a80683b0 0x7f30a806a870 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30cb6b5640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30c41e1eb0 0x7f30c4141190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30cb6b5640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f30c4072fe0 0x7f30c41e1970 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30cb6b5640 1 --2- 172.21.15.62:0/3552301927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00039c0 0x7f30c4072aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:16.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30cb6b5640 1 -- 172.21.15.62:0/3552301927 >> 172.21.15.62:0/3552301927 conn(0x7f30c40b9690 msgr2=0x7f30c40b6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:16.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30cb6b5640 1 -- 172.21.15.62:0/3552301927 shutdown_connections 2024-04-18T10:06:16.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:16.994+0000 7f30cb6b5640 1 -- 172.21.15.62:0/3552301927 wait complete. 2024-04-18T10:06:17.011 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:17.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b7e1a640 1 Processor -- start 2024-04-18T10:06:17.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b7e1a640 1 -- start start 2024-04-18T10:06:17.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b7e1a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b00bbeb0 0x7f18b00b4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b7e1a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f18b00b78e0 0x7f18b00b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b7e1a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18b00bc290 con 0x7f18b00b2a40 2024-04-18T10:06:17.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b7e1a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f18b00b7e90 con 0x7f18b00b78e0 2024-04-18T10:06:17.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b7e1a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f18b00b6740 con 0x7f18b00bbeb0 2024-04-18T10:06:17.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b53b5640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f18b00b2a40 0x7f18b013f8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37138/0 (socket says 172.21.15.62:37138) 2024-04-18T10:06:17.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b53b5640 1 -- 172.21.15.62:0/3996872013 learned_addr learned my addr 172.21.15.62:0/3996872013 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b63b7640 1 --2- 172.21.15.62:0/3996872013 >> v2:172.21.15.62:3300/0 conn(0x7f18b00b78e0 0x7f18b00b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b5bb6640 1 --2- 172.21.15.62:0/3996872013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b00bbeb0 0x7f18b00b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b4bb4640 1 -- 172.21.15.62:0/3996872013 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4252011136 0 0) 0x7f18b00bc290 con 0x7f18b00b2a40 2024-04-18T10:06:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b4bb4640 1 -- 172.21.15.62:0/3996872013 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f189c003530 con 0x7f18b00b2a40 2024-04-18T10:06:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b63b7640 1 -- 172.21.15.62:0/3996872013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b00bbeb0 msgr2=0x7f18b00b4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b4bb4640 1 -- 172.21.15.62:0/3996872013 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2335271337 0 0) 0x7f189c003530 con 0x7f18b00b2a40 2024-04-18T10:06:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b63b7640 1 --2- 172.21.15.62:0/3996872013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b00bbeb0 0x7f18b00b4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b63b7640 1 -- 172.21.15.62:0/3996872013 >> v1:172.21.15.62:6789/0 conn(0x7f18b00b2a40 legacy=0x7f18b013f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.134+0000 7f18b63b7640 1 -- 172.21.15.62:0/3996872013 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b00b63d0 con 0x7f18b00b78e0 2024-04-18T10:06:17.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b63b7640 1 --2- 172.21.15.62:0/3996872013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18b00b78e0 0x7f18b00b46b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009d90 tx=0x7f18a801c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77dbb24411bc16d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b4bb4640 1 -- 172.21.15.62:0/3996872013 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a80ac070 con 0x7f18b00b78e0 2024-04-18T10:06:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b4bb4640 1 -- 172.21.15.62:0/3996872013 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a80a6410 con 0x7f18b00b78e0 2024-04-18T10:06:17.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b4bb4640 1 -- 172.21.15.62:0/3996872013 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0001680 con 0x7f18b00b78e0 2024-04-18T10:06:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 -- 172.21.15.62:0/3996872013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18b00b78e0 msgr2=0x7f18b00b46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 --2- 172.21.15.62:0/3996872013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18b00b78e0 0x7f18b00b46b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009d90 tx=0x7f18a801c800 comp rx=0 tx=0).stop 2024-04-18T10:06:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 -- 172.21.15.62:0/3996872013 shutdown_connections 2024-04-18T10:06:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 --2- 172.21.15.62:0/3996872013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18b00b78e0 0x7f18b00b46b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 --2- 172.21.15.62:0/3996872013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b00bbeb0 0x7f18b00b4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 -- 172.21.15.62:0/3996872013 >> 172.21.15.62:0/3996872013 conn(0x7f18b0067e10 msgr2=0x7f18b00bb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 -- 172.21.15.62:0/3996872013 shutdown_connections 2024-04-18T10:06:17.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 -- 172.21.15.62:0/3996872013 wait complete. 2024-04-18T10:06:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 Processor -- start 2024-04-18T10:06:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 -- start start 2024-04-18T10:06:17.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18b00b2a40 0x7f18b0144ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b00b78e0 0x7f18b01e0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b00bbeb0 0x7f18b006e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f18b0152f70 con 0x7f18b00b2a40 2024-04-18T10:06:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f18b0152df0 con 0x7f18b00b78e0 2024-04-18T10:06:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b7e1a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f18b01530f0 con 0x7f18b00bbeb0 2024-04-18T10:06:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b5bb6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18b00b2a40 0x7f18b0144ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b5bb6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18b00b2a40 0x7f18b0144ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41922/0 (socket says 172.21.15.62:41922) 2024-04-18T10:06:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b5bb6640 1 -- 172.21.15.62:0/4055222375 learned_addr learned my addr 172.21.15.62:0/4055222375 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b63b7640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b00bbeb0 0x7f18b006e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b5bb6640 1 -- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b00bbeb0 msgr2=0x7f18b006e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b5bb6640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b00bbeb0 0x7f18b006e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b53b5640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b00b78e0 0x7f18b01e0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b5bb6640 1 -- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b00b78e0 msgr2=0x7f18b01e0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b5bb6640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b00b78e0 0x7f18b01e0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b5bb6640 1 -- 172.21.15.62:0/4055222375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b01e0190 con 0x7f18b00b2a40 2024-04-18T10:06:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18b5bb6640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18b00b2a40 0x7f18b0144ae0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00e870 tx=0x7f18ac00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.138+0000 7f18a67fc640 1 -- 172.21.15.62:0/4055222375 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac017070 con 0x7f18b00b2a40 2024-04-18T10:06:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.142+0000 7f18a67fc640 1 -- 172.21.15.62:0/4055222375 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18ac00c940 con 0x7f18b00b2a40 2024-04-18T10:06:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.142+0000 7f18b7e1a640 1 -- 172.21.15.62:0/4055222375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b013cc00 con 0x7f18b00b2a40 2024-04-18T10:06:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.142+0000 7f18a67fc640 1 -- 172.21.15.62:0/4055222375 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac00ccf0 con 0x7f18b00b2a40 2024-04-18T10:06:17.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.142+0000 7f18b7e1a640 1 -- 172.21.15.62:0/4055222375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b01e0450 con 0x7f18b00b2a40 2024-04-18T10:06:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.142+0000 7f18b7e1a640 1 -- 172.21.15.62:0/4055222375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b0069230 con 0x7f18b00b2a40 2024-04-18T10:06:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.142+0000 7f18a67fc640 1 -- 172.21.15.62:0/4055222375 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f18ac01a070 con 0x7f18b00b2a40 2024-04-18T10:06:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.142+0000 7f18a67fc640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1894068380 0x7f189406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.142+0000 7f18a67fc640 1 -- 172.21.15.62:0/4055222375 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f18ac0867d0 con 0x7f18b00b2a40 2024-04-18T10:06:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.142+0000 7f18b53b5640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1894068380 0x7f189406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.142+0000 7f18b53b5640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1894068380 0x7f189406a840 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f18a0001ec0 tx=0x7f18a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:17.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.146+0000 7f18a67fc640 1 -- 172.21.15.62:0/4055222375 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f18ac056e20 con 0x7f18b00b2a40 2024-04-18T10:06:17.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.314+0000 7f18b7e1a640 1 -- 172.21.15.62:0/4055222375 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18b013c920 con 0x7f18b00b2a40 2024-04-18T10:06:17.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.314+0000 7f18a67fc640 1 -- 172.21.15.62:0/4055222375 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f18ac056720 con 0x7f18b00b2a40 2024-04-18T10:06:17.314 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:17.314 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:17.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.318+0000 7f18b7e1a640 1 -- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1894068380 msgr2=0x7f189406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.318+0000 7f18b7e1a640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1894068380 0x7f189406a840 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f18a0001ec0 tx=0x7f18a0009040 comp rx=0 tx=0).stop 2024-04-18T10:06:17.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.318+0000 7f18b7e1a640 1 -- 172.21.15.62:0/4055222375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18b00b2a40 msgr2=0x7f18b0144ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.318+0000 7f18b7e1a640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18b00b2a40 0x7f18b0144ae0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00e870 tx=0x7f18ac00ec70 comp rx=0 tx=0).stop 2024-04-18T10:06:17.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.318+0000 7f18b7e1a640 1 -- 172.21.15.62:0/4055222375 shutdown_connections 2024-04-18T10:06:17.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.318+0000 7f18b7e1a640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1894068380 0x7f189406a840 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.318+0000 7f18b7e1a640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b00bbeb0 0x7f18b006e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.318+0000 7f18b7e1a640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b00b78e0 0x7f18b01e0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.318+0000 7f18b7e1a640 1 --2- 172.21.15.62:0/4055222375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f18b00b2a40 0x7f18b0144ae0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.318+0000 7f18b7e1a640 1 -- 172.21.15.62:0/4055222375 >> 172.21.15.62:0/4055222375 conn(0x7f18b0067e10 msgr2=0x7f18b0137500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:17.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.318+0000 7f18b7e1a640 1 -- 172.21.15.62:0/4055222375 shutdown_connections 2024-04-18T10:06:17.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.318+0000 7f18b7e1a640 1 -- 172.21.15.62:0/4055222375 wait complete. 2024-04-18T10:06:17.333 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-18T10:05:56.782649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-18T10:05:57.541623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:17.334 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:17.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.466+0000 7fc1ad08d640 1 Processor -- start 2024-04-18T10:06:17.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.466+0000 7fc1ad08d640 1 -- start start 2024-04-18T10:06:17.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.466+0000 7fc1ad08d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc1a814b190 0x7fc1a8147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.466+0000 7fc1ad08d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1a809b9d0 0x7fc1a8152520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.466+0000 7fc1ad08d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1a814b620 con 0x7fc1a80b3eb0 2024-04-18T10:06:17.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.466+0000 7fc1ad08d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fc1a80b4290 con 0x7fc1a814b190 2024-04-18T10:06:17.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.466+0000 7fc1ad08d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a814c100 con 0x7fc1a809b9d0 2024-04-18T10:06:17.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a659c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc1a814b190 0x7fc1a8147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a759e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1a809b9d0 0x7fc1a8152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a659c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc1a814b190 0x7fc1a8147ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.62:55340/0 (socket says 172.21.15.62:55340) 2024-04-18T10:06:17.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a659c640 1 -- 172.21.15.62:0/117671102 learned_addr learned my addr 172.21.15.62:0/117671102 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:17.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a6d9d640 1 --1- 172.21.15.62:0/117671102 >> v1:172.21.15.145:6789/0 conn(0x7fc1a80b3eb0 0x7fc1a80bdd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.62:46368/0 (socket says 172.21.15.62:46368) 2024-04-18T10:06:17.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a659c640 1 -- 172.21.15.62:0/117671102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1a809b9d0 msgr2=0x7fc1a8152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:17.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a659c640 1 --2- 172.21.15.62:0/117671102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1a809b9d0 0x7fc1a8152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a659c640 1 -- 172.21.15.62:0/117671102 >> v1:172.21.15.145:6789/0 conn(0x7fc1a80b3eb0 legacy=0x7fc1a80bdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a659c640 1 -- 172.21.15.62:0/117671102 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a814b900 con 0x7fc1a814b190 2024-04-18T10:06:17.470 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a659c640 1 --2- 172.21.15.62:0/117671102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1a814b190 0x7fc1a8147ba0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc19c009870 tx=0x7fc19c01c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d0881d4fee74f11 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:17.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a5d9b640 1 -- 172.21.15.62:0/117671102 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c0b0070 con 0x7fc1a814b190 2024-04-18T10:06:17.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a5d9b640 1 -- 172.21.15.62:0/117671102 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc19c0ab600 con 0x7fc1a814b190 2024-04-18T10:06:17.471 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.470+0000 7fc1a5d9b640 1 -- 172.21.15.62:0/117671102 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c0a4e60 con 0x7fc1a814b190 2024-04-18T10:06:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 -- 172.21.15.62:0/117671102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1a814b190 msgr2=0x7fc1a8147ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 --2- 172.21.15.62:0/117671102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1a814b190 0x7fc1a8147ba0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc19c009870 tx=0x7fc19c01c4d0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 -- 172.21.15.62:0/117671102 shutdown_connections 2024-04-18T10:06:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 --2- 172.21.15.62:0/117671102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1a809b9d0 0x7fc1a8152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 --2- 172.21.15.62:0/117671102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1a814b190 0x7fc1a8147ba0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 -- 172.21.15.62:0/117671102 >> 172.21.15.62:0/117671102 conn(0x7fc1a8067e10 msgr2=0x7fc1a80bd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 -- 172.21.15.62:0/117671102 shutdown_connections 2024-04-18T10:06:17.472 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 -- 172.21.15.62:0/117671102 wait complete. 2024-04-18T10:06:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 Processor -- start 2024-04-18T10:06:17.473 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 -- start start 2024-04-18T10:06:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1a80b3eb0 0x7fc1a809f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1a81e0e20 0x7fc1a81e1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1a81e4100 0x7fc1a81e4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a8155610 con 0x7fc1a81e0e20 2024-04-18T10:06:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fc1a8155490 con 0x7fc1a81e4100 2024-04-18T10:06:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc1a8155790 con 0x7fc1a80b3eb0 2024-04-18T10:06:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1a659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1a81e0e20 0x7fc1a81e1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1a659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1a81e0e20 0x7fc1a81e1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41930/0 (socket says 172.21.15.62:41930) 2024-04-18T10:06:17.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1a659c640 1 -- 172.21.15.62:0/920291183 learned_addr learned my addr 172.21.15.62:0/920291183 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1a759e640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1a81e4100 0x7fc1a81e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1a659c640 1 -- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1a80b3eb0 msgr2=0x7fc1a809f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1a659c640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1a80b3eb0 0x7fc1a809f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1a659c640 1 -- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1a81e4100 msgr2=0x7fc1a81e4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1a659c640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1a81e4100 0x7fc1a81e4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1a659c640 1 -- 172.21.15.62:0/920291183 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1a809f6e0 con 0x7fc1a81e0e20 2024-04-18T10:06:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1a659c640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1a81e0e20 0x7fc1a81e1280 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0ae040 tx=0x7fc19c01ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc18f7fe640 1 -- 172.21.15.62:0/920291183 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c0ba020 con 0x7fc1a81e0e20 2024-04-18T10:06:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc18f7fe640 1 -- 172.21.15.62:0/920291183 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc19c0a4e60 con 0x7fc1a81e0e20 2024-04-18T10:06:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 -- 172.21.15.62:0/920291183 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1a81e17c0 con 0x7fc1a81e0e20 2024-04-18T10:06:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc18f7fe640 1 -- 172.21.15.62:0/920291183 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc19c0b0040 con 0x7fc1a81e0e20 2024-04-18T10:06:17.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.474+0000 7fc1ad08d640 1 -- 172.21.15.62:0/920291183 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1a80baa70 con 0x7fc1a81e0e20 2024-04-18T10:06:17.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.478+0000 7fc1ad08d640 1 -- 172.21.15.62:0/920291183 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1a8068c00 con 0x7fc1a81e0e20 2024-04-18T10:06:17.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.478+0000 7fc18f7fe640 1 -- 172.21.15.62:0/920291183 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc19c0bd020 con 0x7fc1a81e0e20 2024-04-18T10:06:17.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.478+0000 7fc18f7fe640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc194068380 0x7fc19406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.478+0000 7fc18f7fe640 1 -- 172.21.15.62:0/920291183 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc19c120d10 con 0x7fc1a81e0e20 2024-04-18T10:06:17.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.478+0000 7fc1a6d9d640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc194068380 0x7fc19406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.478+0000 7fc1a6d9d640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc194068380 0x7fc19406a840 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc198000f70 tx=0x7fc198007a10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:17.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.482+0000 7fc18f7fe640 1 -- 172.21.15.62:0/920291183 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc19c0c4030 con 0x7fc1a81e0e20 2024-04-18T10:06:17.647 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 -- 172.21.15.62:0/920291183 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc1a809b2d0 con 0x7fc1a81e0e20 2024-04-18T10:06:17.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc18f7fe640 1 -- 172.21.15.62:0/920291183 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fc19c016150 con 0x7fc1a81e0e20 2024-04-18T10:06:17.648 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 -- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc194068380 msgr2=0x7fc19406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc194068380 0x7fc19406a840 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc198000f70 tx=0x7fc198007a10 comp rx=0 tx=0).stop 2024-04-18T10:06:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 -- 172.21.15.62:0/920291183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1a81e0e20 msgr2=0x7fc1a81e1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1a81e0e20 0x7fc1a81e1280 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0ae040 tx=0x7fc19c01ad10 comp rx=0 tx=0).stop 2024-04-18T10:06:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 -- 172.21.15.62:0/920291183 shutdown_connections 2024-04-18T10:06:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc194068380 0x7fc19406a840 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1a81e4100 0x7fc1a81e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc1a81e0e20 0x7fc1a81e1280 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 --2- 172.21.15.62:0/920291183 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1a80b3eb0 0x7fc1a809f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 -- 172.21.15.62:0/920291183 >> 172.21.15.62:0/920291183 conn(0x7fc1a8067e10 msgr2=0x7fc1a80bb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 -- 172.21.15.62:0/920291183 shutdown_connections 2024-04-18T10:06:17.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.650+0000 7fc1ad08d640 1 -- 172.21.15.62:0/920291183 wait complete. 2024-04-18T10:06:17.667 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-18T10:06:17.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccf904640 1 Processor -- start 2024-04-18T10:06:17.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccf904640 1 -- start start 2024-04-18T10:06:17.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccf904640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3cc81454c0 0x7f3cc80732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccf904640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cc80bbe40 con 0x7f3cc80ba5c0 2024-04-18T10:06:17.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccf904640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cc80bb360 con 0x7f3cc80ba9a0 2024-04-18T10:06:17.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccf904640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f3cc8073820 con 0x7f3cc81454c0 2024-04-18T10:06:17.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccd6a0640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3cc80ba5c0 0x7f3cc8147df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37152/0 (socket says 172.21.15.62:37152) 2024-04-18T10:06:17.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccd6a0640 1 -- 172.21.15.62:0/4286506051 learned_addr learned my addr 172.21.15.62:0/4286506051 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:17.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3848292876 0 0) 0x7f3cc80bbe40 con 0x7f3cc80ba5c0 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccdea1640 1 --2- 172.21.15.62:0/4286506051 >> v2:172.21.15.145:3300/0 conn(0x7f3cc81454c0 0x7f3cc80732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3cb4003530 con 0x7f3cc80ba5c0 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1934711103 0 0) 0x7f3cb4003530 con 0x7f3cc80ba5c0 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 >> v2:172.21.15.145:3300/0 conn(0x7f3cc81454c0 msgr2=0x7f3cc80732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 --2- 172.21.15.62:0/4286506051 >> v2:172.21.15.145:3300/0 conn(0x7f3cc81454c0 0x7f3cc80732e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 >> v1:172.21.15.145:6789/0 conn(0x7f3cc80ba9a0 legacy=0x7f3cc8144db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc80bb640 con 0x7f3cc80ba5c0 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f3cc0001a00 con 0x7f3cc80ba5c0 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 >> v1:172.21.15.62:6789/0 conn(0x7f3cc80ba5c0 legacy=0x7f3cc8147df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cb40039c0 0x7f3cb4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cb4004810 0x7f3cb4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cb40051d0 0x7f3cb400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc8073820 con 0x7f3cb40039c0 2024-04-18T10:06:17.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb400c170 con 0x7f3cb40051d0 2024-04-18T10:06:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb400c370 con 0x7f3cb4004810 2024-04-18T10:06:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccd6a0640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cb40039c0 0x7f3cb4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccdea1640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cb40051d0 0x7f3cb400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccd6a0640 1 -- 172.21.15.62:0/4286506051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cb4004810 msgr2=0x7f3cb4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccce9f640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cb4004810 0x7f3cb4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccd6a0640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cb4004810 0x7f3cb4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccd6a0640 1 -- 172.21.15.62:0/4286506051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cb40051d0 msgr2=0x7f3cb400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccd6a0640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cb40051d0 0x7f3cb400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccd6a0640 1 -- 172.21.15.62:0/4286506051 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb4003530 con 0x7f3cb40039c0 2024-04-18T10:06:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccdea1640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cb40051d0 0x7f3cb400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccce9f640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cb4004810 0x7f3cb4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:17.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.790+0000 7f3ccdea1640 1 -- 172.21.15.62:0/4286506051 reap_dead start 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3ccd6a0640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cb40039c0 0x7f3cb4003de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000a840 tx=0x7f3cc001d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f6e66d65495072 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc000a460 con 0x7f3cb40039c0 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cc00ad650 con 0x7f3cb40039c0 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3cbffff640 1 -- 172.21.15.62:0/4286506051 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc00ada40 con 0x7f3cb40039c0 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3ccf904640 1 -- 172.21.15.62:0/4286506051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cb40039c0 msgr2=0x7f3cb4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3ccf904640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cb40039c0 0x7f3cb4003de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000a840 tx=0x7f3cc001d360 comp rx=0 tx=0).stop 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3ccf904640 1 -- 172.21.15.62:0/4286506051 shutdown_connections 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3ccf904640 1 --2- 172.21.15.62:0/4286506051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cb40039c0 0x7f3cb4003de0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3ccf904640 1 -- 172.21.15.62:0/4286506051 >> 172.21.15.62:0/4286506051 conn(0x7f3cc80a0f00 msgr2=0x7f3cc809aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3ccf904640 1 -- 172.21.15.62:0/4286506051 shutdown_connections 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3ccf904640 1 -- 172.21.15.62:0/4286506051 wait complete. 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3ccf904640 1 Processor -- start 2024-04-18T10:06:17.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.794+0000 7f3ccf904640 1 -- start start 2024-04-18T10:06:17.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccf904640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cb40039c0 0x7f3cc81e0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccf904640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cc806e520 0x7f3cc814a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccf904640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cc8141800 0x7f3cc81d6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccf904640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc814b6f0 con 0x7f3cc806e520 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccf904640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc814b570 con 0x7f3cb40039c0 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccf904640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3cc814b870 con 0x7f3cc8141800 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccce9f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cc806e520 0x7f3cc814a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccce9f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cc806e520 0x7f3cc814a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41952/0 (socket says 172.21.15.62:41952) 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccce9f640 1 -- 172.21.15.62:0/1004114558 learned_addr learned my addr 172.21.15.62:0/1004114558 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccdea1640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cc8141800 0x7f3cc81d6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccd6a0640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cb40039c0 0x7f3cc81e0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccce9f640 1 -- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cc8141800 msgr2=0x7f3cc81d6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccce9f640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cc8141800 0x7f3cc81d6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccce9f640 1 -- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cb40039c0 msgr2=0x7f3cc81e0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccce9f640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cb40039c0 0x7f3cc81e0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccce9f640 1 -- 172.21.15.62:0/1004114558 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc81494e0 con 0x7f3cc806e520 2024-04-18T10:06:17.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccd6a0640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cb40039c0 0x7f3cc81e0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:17.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccce9f640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cc806e520 0x7f3cc814a580 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3cb800d4a0 tx=0x7f3cb800d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:17.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3cbdffb640 1 -- 172.21.15.62:0/1004114558 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8019070 con 0x7f3cc806e520 2024-04-18T10:06:17.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3cbdffb640 1 -- 172.21.15.62:0/1004114558 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cb80054a0 con 0x7f3cc806e520 2024-04-18T10:06:17.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccf904640 1 -- 172.21.15.62:0/1004114558 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cc806ea00 con 0x7f3cc806e520 2024-04-18T10:06:17.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3cbdffb640 1 -- 172.21.15.62:0/1004114558 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8007270 con 0x7f3cc806e520 2024-04-18T10:06:17.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccf904640 1 -- 172.21.15.62:0/1004114558 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cc8149160 con 0x7f3cc806e520 2024-04-18T10:06:17.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccf904640 1 -- 172.21.15.62:0/1004114558 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cc8069230 con 0x7f3cc806e520 2024-04-18T10:06:17.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3cbdffb640 1 -- 172.21.15.62:0/1004114558 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3cb801c070 con 0x7f3cc806e520 2024-04-18T10:06:17.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3cbdffb640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3cac0683b0 0x7f3cac06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:17.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3cbdffb640 1 -- 172.21.15.62:0/1004114558 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3cb8086030 con 0x7f3cc806e520 2024-04-18T10:06:17.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.798+0000 7f3ccd6a0640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3cac0683b0 0x7f3cac06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:17.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.802+0000 7f3ccd6a0640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3cac0683b0 0x7f3cac06a870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3cc00ad000 tx=0x7f3cc8070b00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:17.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.802+0000 7f3cbdffb640 1 -- 172.21.15.62:0/1004114558 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3cb8056740 con 0x7f3cc806e520 2024-04-18T10:06:17.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.966+0000 7f3ccf904640 1 -- 172.21.15.62:0/1004114558 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3cc81448f0 con 0x7f3cc806e520 2024-04-18T10:06:17.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3cbdffb640 1 -- 172.21.15.62:0/1004114558 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f3cb8056040 con 0x7f3cc806e520 2024-04-18T10:06:17.967 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-18T10:06:17.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3ccf904640 1 -- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3cac0683b0 msgr2=0x7f3cac06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3ccf904640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3cac0683b0 0x7f3cac06a870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3cc00ad000 tx=0x7f3cc8070b00 comp rx=0 tx=0).stop 2024-04-18T10:06:17.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3ccf904640 1 -- 172.21.15.62:0/1004114558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cc806e520 msgr2=0x7f3cc814a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:17.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3ccf904640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cc806e520 0x7f3cc814a580 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3cb800d4a0 tx=0x7f3cb800d960 comp rx=0 tx=0).stop 2024-04-18T10:06:17.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3ccf904640 1 -- 172.21.15.62:0/1004114558 shutdown_connections 2024-04-18T10:06:17.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3ccf904640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3cac0683b0 0x7f3cac06a870 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3ccf904640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3cc8141800 0x7f3cc81d6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3ccf904640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3cc806e520 0x7f3cc814a580 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3ccf904640 1 --2- 172.21.15.62:0/1004114558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3cb40039c0 0x7f3cc81e0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:17.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3ccf904640 1 -- 172.21.15.62:0/1004114558 >> 172.21.15.62:0/1004114558 conn(0x7f3cc80a0f00 msgr2=0x7f3cc80b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:17.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3ccf904640 1 -- 172.21.15.62:0/1004114558 shutdown_connections 2024-04-18T10:06:17.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:17.970+0000 7f3ccf904640 1 -- 172.21.15.62:0/1004114558 wait complete. 2024-04-18T10:06:17.986 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-18T10:06:18.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.114+0000 7f9cb4000640 1 Processor -- start 2024-04-18T10:06:18.114 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.114+0000 7f9cb4000640 1 -- start start 2024-04-18T10:06:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb4000640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9cac06d500 0x7f9cac0aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb4000640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cac0ae680 con 0x7f9cac09b160 2024-04-18T10:06:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb4000640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cac06df00 con 0x7f9cac0af330 2024-04-18T10:06:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb4000640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f9cac06d8e0 con 0x7f9cac06d500 2024-04-18T10:06:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb259d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9cac09b160 0x7f9cac156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37166/0 (socket says 172.21.15.62:37166) 2024-04-18T10:06:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb259d640 1 -- 172.21.15.62:0/3627765143 learned_addr learned my addr 172.21.15.62:0/3627765143 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb1d9c640 1 --2- 172.21.15.62:0/3627765143 >> v2:172.21.15.145:3300/0 conn(0x7f9cac06d500 0x7f9cac0aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2343630155 0 0) 0x7f9ca40014a0 con 0x7f9cac09b160 2024-04-18T10:06:18.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c98003530 con 0x7f9cac09b160 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3040263238 0 0) 0x7f9c98003530 con 0x7f9cac09b160 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 >> v2:172.21.15.145:3300/0 conn(0x7f9cac06d500 msgr2=0x7f9cac0aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 --2- 172.21.15.62:0/3627765143 >> v2:172.21.15.145:3300/0 conn(0x7f9cac06d500 0x7f9cac0aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 >> v1:172.21.15.145:6789/0 conn(0x7f9cac0af330 legacy=0x7f9cac09aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cac0ae960 con 0x7f9cac09b160 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f9ca4001bd0 con 0x7f9cac09b160 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 >> v1:172.21.15.62:6789/0 conn(0x7f9cac09b160 legacy=0x7f9cac156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 --2- 172.21.15.62:0/3627765143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980039c0 0x7f9c98003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 --2- 172.21.15.62:0/3627765143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c98004810 0x7f9c98004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 --2- 172.21.15.62:0/3627765143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c980052f0 0x7f9c9800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac06d8e0 con 0x7f9c980052f0 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9800c1c0 con 0x7f9c98004810 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f9c9800c3c0 con 0x7f9c980039c0 2024-04-18T10:06:18.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb259d640 1 --2- 172.21.15.62:0/3627765143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c980052f0 0x7f9c9800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb159b640 1 --2- 172.21.15.62:0/3627765143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c98004810 0x7f9c98004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb1d9c640 1 --2- 172.21.15.62:0/3627765143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980039c0 0x7f9c98003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb259d640 1 -- 172.21.15.62:0/3627765143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980039c0 msgr2=0x7f9c98003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb259d640 1 --2- 172.21.15.62:0/3627765143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980039c0 0x7f9c98003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb259d640 1 -- 172.21.15.62:0/3627765143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c98004810 msgr2=0x7f9c98004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb259d640 1 --2- 172.21.15.62:0/3627765143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c98004810 0x7f9c98004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb259d640 1 -- 172.21.15.62:0/3627765143 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c98003530 con 0x7f9c980052f0 2024-04-18T10:06:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb259d640 1 -- 172.21.15.62:0/3627765143 reap_dead start 2024-04-18T10:06:18.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb259d640 1 --2- 172.21.15.62:0/3627765143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c980052f0 0x7f9c9800b9d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9ca400aa00 tx=0x7f9ca401d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72458bf69ee5e7e server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca400a650 con 0x7f9c980052f0 2024-04-18T10:06:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ca40ad650 con 0x7f9c980052f0 2024-04-18T10:06:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.118+0000 7f9cb0d9a640 1 -- 172.21.15.62:0/3627765143 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40ada40 con 0x7f9c980052f0 2024-04-18T10:06:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 -- 172.21.15.62:0/3627765143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c980052f0 msgr2=0x7f9c9800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 --2- 172.21.15.62:0/3627765143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c980052f0 0x7f9c9800b9d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9ca400aa00 tx=0x7f9ca401d340 comp rx=0 tx=0).stop 2024-04-18T10:06:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 -- 172.21.15.62:0/3627765143 shutdown_connections 2024-04-18T10:06:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 --2- 172.21.15.62:0/3627765143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9c980052f0 0x7f9c9800b9d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 -- 172.21.15.62:0/3627765143 >> 172.21.15.62:0/3627765143 conn(0x7f9cac0b9690 msgr2=0x7f9cac0b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 -- 172.21.15.62:0/3627765143 shutdown_connections 2024-04-18T10:06:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 -- 172.21.15.62:0/3627765143 wait complete. 2024-04-18T10:06:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 Processor -- start 2024-04-18T10:06:18.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 -- start start 2024-04-18T10:06:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980052f0 0x7f9cac148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cac149d30 0x7f9cac1e5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cac1e5660 0x7f9cac1e8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac0b57b0 con 0x7f9cac149d30 2024-04-18T10:06:18.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac0b5630 con 0x7f9cac1e5660 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f9cac0b5a30 con 0x7f9c980052f0 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb159b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cac149d30 0x7f9cac1e5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb159b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cac149d30 0x7f9cac1e5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41966/0 (socket says 172.21.15.62:41966) 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb159b640 1 -- 172.21.15.62:0/3033894562 learned_addr learned my addr 172.21.15.62:0/3033894562 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb259d640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cac1e5660 0x7f9cac1e8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb1d9c640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980052f0 0x7f9cac148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb159b640 1 -- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980052f0 msgr2=0x7f9cac148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb159b640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980052f0 0x7f9cac148e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb159b640 1 -- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cac1e5660 msgr2=0x7f9cac1e8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb159b640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cac1e5660 0x7f9cac1e8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb159b640 1 -- 172.21.15.62:0/3033894562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cac1e5ae0 con 0x7f9cac149d30 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb259d640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cac1e5660 0x7f9cac1e8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb159b640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cac149d30 0x7f9cac1e5120 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c00f5f0 tx=0x7f9c9c00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:18.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9ca27fc640 1 -- 172.21.15.62:0/3033894562 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c0052e0 con 0x7f9cac149d30 2024-04-18T10:06:18.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9ca27fc640 1 -- 172.21.15.62:0/3033894562 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c9c018070 con 0x7f9cac149d30 2024-04-18T10:06:18.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 -- 172.21.15.62:0/3033894562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cac147a70 con 0x7f9cac149d30 2024-04-18T10:06:18.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9ca27fc640 1 -- 172.21.15.62:0/3033894562 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c003820 con 0x7f9cac149d30 2024-04-18T10:06:18.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.122+0000 7f9cb4000640 1 -- 172.21.15.62:0/3033894562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cac148730 con 0x7f9cac149d30 2024-04-18T10:06:18.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.126+0000 7f9cb4000640 1 -- 172.21.15.62:0/3033894562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cac068c00 con 0x7f9cac149d30 2024-04-18T10:06:18.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.126+0000 7f9ca27fc640 1 -- 172.21.15.62:0/3033894562 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9c9c0039c0 con 0x7f9cac149d30 2024-04-18T10:06:18.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.126+0000 7f9ca27fc640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9c90068380 0x7f9c9006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.126+0000 7f9ca27fc640 1 -- 172.21.15.62:0/3033894562 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9c9c0866b0 con 0x7f9cac149d30 2024-04-18T10:06:18.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.126+0000 7f9cb1d9c640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9c90068380 0x7f9c9006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.126+0000 7f9cb1d9c640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9c90068380 0x7f9c9006a840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8009850 tx=0x7f9ca8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:18.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.130+0000 7f9ca27fc640 1 -- 172.21.15.62:0/3033894562 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9cac1e5ae0 con 0x7f9cac149d30 2024-04-18T10:06:18.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.282+0000 7f9cb4000640 1 -- 172.21.15.62:0/3033894562 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9cac09e660 con 0x7f9cac149d30 2024-04-18T10:06:18.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.282+0000 7f9ca27fc640 1 -- 172.21.15.62:0/3033894562 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f9c9c056d00 con 0x7f9cac149d30 2024-04-18T10:06:18.283 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-04-18T10:06:18.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.286+0000 7f9cb4000640 1 -- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9c90068380 msgr2=0x7f9c9006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.286+0000 7f9cb4000640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9c90068380 0x7f9c9006a840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8009850 tx=0x7f9ca8008040 comp rx=0 tx=0).stop 2024-04-18T10:06:18.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.286+0000 7f9cb4000640 1 -- 172.21.15.62:0/3033894562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cac149d30 msgr2=0x7f9cac1e5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.286+0000 7f9cb4000640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cac149d30 0x7f9cac1e5120 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c00f5f0 tx=0x7f9c9c00fab0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.286+0000 7f9cb4000640 1 -- 172.21.15.62:0/3033894562 shutdown_connections 2024-04-18T10:06:18.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.286+0000 7f9cb4000640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9c90068380 0x7f9c9006a840 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.286+0000 7f9cb4000640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cac1e5660 0x7f9cac1e8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.286+0000 7f9cb4000640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cac149d30 0x7f9cac1e5120 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.286+0000 7f9cb4000640 1 --2- 172.21.15.62:0/3033894562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980052f0 0x7f9cac148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.286+0000 7f9cb4000640 1 -- 172.21.15.62:0/3033894562 >> 172.21.15.62:0/3033894562 conn(0x7f9cac0b9690 msgr2=0x7f9cac14e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:18.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.286+0000 7f9cb4000640 1 -- 172.21.15.62:0/3033894562 shutdown_connections 2024-04-18T10:06:18.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.286+0000 7f9cb4000640 1 -- 172.21.15.62:0/3033894562 wait complete. 2024-04-18T10:06:18.301 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:18.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.410+0000 7f5705075640 1 Processor -- start 2024-04-18T10:06:18.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f5705075640 1 -- start start 2024-04-18T10:06:18.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f5705075640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57000b3e20 0x7f57000afae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f5705075640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5700144390 con 0x7f570006d470 2024-04-18T10:06:18.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f5705075640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570009c4d0 con 0x7f570009cc40 2024-04-18T10:06:18.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f5705075640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f57000b5b40 con 0x7f57000b3e20 2024-04-18T10:06:18.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fed9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f570006d470 0x7f5700144b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37176/0 (socket says 172.21.15.62:37176) 2024-04-18T10:06:18.412 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fed9d640 1 -- 172.21.15.62:0/3810294112 learned_addr learned my addr 172.21.15.62:0/3810294112 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:18.413 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fe59c640 1 --2- 172.21.15.62:0/3810294112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57000b3e20 0x7f57000afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2485645674 0 0) 0x7f5700144390 con 0x7f570006d470 2024-04-18T10:06:18.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56e8003530 con 0x7f570006d470 2024-04-18T10:06:18.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4242241632 0 0) 0x7f56e8003530 con 0x7f570006d470 2024-04-18T10:06:18.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57000b3e20 msgr2=0x7f57000afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:18.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 --2- 172.21.15.62:0/3810294112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57000b3e20 0x7f57000afae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 >> v1:172.21.15.145:6789/0 conn(0x7f570009cc40 legacy=0x7f570014ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570006d850 con 0x7f570006d470 2024-04-18T10:06:18.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f56f4001680 con 0x7f570006d470 2024-04-18T10:06:18.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fe59c640 1 --2- 172.21.15.62:0/3810294112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57000b3e20 0x7f57000afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:18.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 >> v1:172.21.15.62:6789/0 conn(0x7f570006d470 legacy=0x7f5700144b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 --2- 172.21.15.62:0/3810294112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56e80039f0 0x7f56e8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.414 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 --2- 172.21.15.62:0/3810294112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57000b3e20 0x7f56e8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 --2- 172.21.15.62:0/3810294112 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56e8004d50 0x7f56e80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f56e800bbd0 con 0x7f56e80039f0 2024-04-18T10:06:18.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f56e800bd80 con 0x7f56e8004d50 2024-04-18T10:06:18.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f56e800bf80 con 0x7f57000b3e20 2024-04-18T10:06:18.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fed9d640 1 --2- 172.21.15.62:0/3810294112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56e80039f0 0x7f56e8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fed9d640 1 -- 172.21.15.62:0/3810294112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57000b3e20 msgr2=0x7f56e8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:18.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fed9d640 1 --2- 172.21.15.62:0/3810294112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f57000b3e20 0x7f56e8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fed9d640 1 -- 172.21.15.62:0/3810294112 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56e8004d50 msgr2=0x7f56e80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:18.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fed9d640 1 --2- 172.21.15.62:0/3810294112 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56e8004d50 0x7f56e80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.415 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.414+0000 7f56fed9d640 1 -- 172.21.15.62:0/3810294112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e8003530 con 0x7f56e80039f0 2024-04-18T10:06:18.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fed9d640 1 --2- 172.21.15.62:0/3810294112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56e80039f0 0x7f56e8003e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f56f400c210 tx=0x7f56f401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1364a2669a31baac server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:18.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4007d40 con 0x7f56e80039f0 2024-04-18T10:06:18.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f40a9070 con 0x7f56e80039f0 2024-04-18T10:06:18.416 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fdd9b640 1 -- 172.21.15.62:0/3810294112 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4005d40 con 0x7f56e80039f0 2024-04-18T10:06:18.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 -- 172.21.15.62:0/3810294112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56e80039f0 msgr2=0x7f56e8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 --2- 172.21.15.62:0/3810294112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56e80039f0 0x7f56e8003e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f56f400c210 tx=0x7f56f401cca0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56ff59e640 1 -- 172.21.15.62:0/3810294112 reap_dead start 2024-04-18T10:06:18.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 -- 172.21.15.62:0/3810294112 shutdown_connections 2024-04-18T10:06:18.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 -- 172.21.15.62:0/3810294112 >> 172.21.15.62:0/3810294112 conn(0x7f570006c0d0 msgr2=0x7f57000b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:18.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 -- 172.21.15.62:0/3810294112 shutdown_connections 2024-04-18T10:06:18.417 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 -- 172.21.15.62:0/3810294112 wait complete. 2024-04-18T10:06:18.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 Processor -- start 2024-04-18T10:06:18.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 -- start start 2024-04-18T10:06:18.419 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f570013f360 0x7f570009b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f570014a5b0 0x7f57001e2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f570013f740 0x7f57000b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f57001502c0 con 0x7f570013f360 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f5700150140 con 0x7f570013f740 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f5700150440 con 0x7f570014a5b0 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f570013f360 0x7f570009b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fed9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f570013f360 0x7f570009b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41976/0 (socket says 172.21.15.62:41976) 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fed9d640 1 -- 172.21.15.62:0/1764196886 learned_addr learned my addr 172.21.15.62:0/1764196886 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fe59c640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f570014a5b0 0x7f57001e2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56ff59e640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f570013f740 0x7f57000b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fed9d640 1 -- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f570014a5b0 msgr2=0x7f57001e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fed9d640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f570014a5b0 0x7f57001e2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fed9d640 1 -- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f570013f740 msgr2=0x7f57000b0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fed9d640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f570013f740 0x7f57000b0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.420 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fed9d640 1 -- 172.21.15.62:0/1764196886 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570014d000 con 0x7f570013f360 2024-04-18T10:06:18.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56fed9d640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f570013f360 0x7f570009b0d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f56f4001980 tx=0x7f56e8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:18.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56e77fe640 1 -- 172.21.15.62:0/1764196886 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f40048b0 con 0x7f570013f360 2024-04-18T10:06:18.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56e77fe640 1 -- 172.21.15.62:0/1764196886 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f4004a70 con 0x7f570013f360 2024-04-18T10:06:18.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 -- 172.21.15.62:0/1764196886 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570014d2e0 con 0x7f570013f360 2024-04-18T10:06:18.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f56e77fe640 1 -- 172.21.15.62:0/1764196886 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f4004e60 con 0x7f570013f360 2024-04-18T10:06:18.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.418+0000 7f5705075640 1 -- 172.21.15.62:0/1764196886 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57000b1c20 con 0x7f570013f360 2024-04-18T10:06:18.421 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.422+0000 7f5705075640 1 -- 172.21.15.62:0/1764196886 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5700068b90 con 0x7f570013f360 2024-04-18T10:06:18.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.422+0000 7f56e77fe640 1 -- 172.21.15.62:0/1764196886 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f56f4007180 con 0x7f570013f360 2024-04-18T10:06:18.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.422+0000 7f56e77fe640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56ec068380 0x7f56ec06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.422+0000 7f56e77fe640 1 -- 172.21.15.62:0/1764196886 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f56f4120120 con 0x7f570013f360 2024-04-18T10:06:18.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.422+0000 7f56fe59c640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56ec068380 0x7f56ec06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.422 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.422+0000 7f56fe59c640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56ec068380 0x7f56ec06a840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f570014a9d0 tx=0x7f56f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:18.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.426+0000 7f56e77fe640 1 -- 172.21.15.62:0/1764196886 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f56f40169b0 con 0x7f570013f360 2024-04-18T10:06:18.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.578+0000 7f5705075640 1 -- 172.21.15.62:0/1764196886 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f570014d4b0 con 0x7f570013f360 2024-04-18T10:06:18.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.582+0000 7f56e77fe640 1 -- 172.21.15.62:0/1764196886 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f56f40f0770 con 0x7f570013f360 2024-04-18T10:06:18.582 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:18.582 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:18.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.586+0000 7f5705075640 1 -- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56ec068380 msgr2=0x7f56ec06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.586+0000 7f5705075640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56ec068380 0x7f56ec06a840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f570014a9d0 tx=0x7f56f0008040 comp rx=0 tx=0).stop 2024-04-18T10:06:18.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.586+0000 7f5705075640 1 -- 172.21.15.62:0/1764196886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f570013f360 msgr2=0x7f570009b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.586+0000 7f5705075640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f570013f360 0x7f570009b0d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f56f4001980 tx=0x7f56e8005840 comp rx=0 tx=0).stop 2024-04-18T10:06:18.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.586+0000 7f5705075640 1 -- 172.21.15.62:0/1764196886 shutdown_connections 2024-04-18T10:06:18.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.586+0000 7f5705075640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56ec068380 0x7f56ec06a840 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.586+0000 7f5705075640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f570013f740 0x7f57000b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.586+0000 7f5705075640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f570014a5b0 0x7f57001e2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.586+0000 7f5705075640 1 --2- 172.21.15.62:0/1764196886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f570013f360 0x7f570009b0d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.586+0000 7f5705075640 1 -- 172.21.15.62:0/1764196886 >> 172.21.15.62:0/1764196886 conn(0x7f570006c0d0 msgr2=0x7f57000aedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:18.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.586+0000 7f5705075640 1 -- 172.21.15.62:0/1764196886 shutdown_connections 2024-04-18T10:06:18.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.586+0000 7f5705075640 1 -- 172.21.15.62:0/1764196886 wait complete. 2024-04-18T10:06:18.599 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-18T10:05:56.782649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-18T10:05:57.541623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:18.599 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:18.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.726+0000 7efdc2518640 1 Processor -- start 2024-04-18T10:06:18.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.726+0000 7efdc2518640 1 -- start start 2024-04-18T10:06:18.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdc2518640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efdbc0b01a0 0x7efdbc06d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdc2518640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdbc0b0580 con 0x7efdbc13f090 2024-04-18T10:06:18.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdc2518640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdbc0b59d0 con 0x7efdbc0ae4e0 2024-04-18T10:06:18.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdc2518640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7efdbc0ae970 con 0x7efdbc0b01a0 2024-04-18T10:06:18.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdc0ab5640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7efdbc13f090 0x7efdbc14f430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37192/0 (socket says 172.21.15.62:37192) 2024-04-18T10:06:18.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdc0ab5640 1 -- 172.21.15.62:0/1909960847 learned_addr learned my addr 172.21.15.62:0/1909960847 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:18.730 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbbfff640 1 --2- 172.21.15.62:0/1909960847 >> v2:172.21.15.145:3300/0 conn(0x7efdbc0b01a0 0x7efdbc06d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3731821325 0 0) 0x7efdbc0b0580 con 0x7efdbc13f090 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efda4003530 con 0x7efdbc13f090 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2753359407 0 0) 0x7efda4003530 con 0x7efdbc13f090 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 >> v2:172.21.15.145:3300/0 conn(0x7efdbc0b01a0 msgr2=0x7efdbc06d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 --2- 172.21.15.62:0/1909960847 >> v2:172.21.15.145:3300/0 conn(0x7efdbc0b01a0 0x7efdbc06d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 >> v1:172.21.15.145:6789/0 conn(0x7efdbc0ae4e0 legacy=0x7efdbc13e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdbc0b53a0 con 0x7efdbc13f090 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7efdb4001a00 con 0x7efdbc13f090 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 >> v1:172.21.15.62:6789/0 conn(0x7efdbc13f090 legacy=0x7efdbc14f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 --2- 172.21.15.62:0/1909960847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efda40039c0 0x7efda4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 --2- 172.21.15.62:0/1909960847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efda4004810 0x7efda4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 --2- 172.21.15.62:0/1909960847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efda40052f0 0x7efda400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc0ae970 con 0x7efda40052f0 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7efda400c1c0 con 0x7efda40039c0 2024-04-18T10:06:18.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7efda400c3c0 con 0x7efda4004810 2024-04-18T10:06:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdc0ab5640 1 --2- 172.21.15.62:0/1909960847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efda40052f0 0x7efda400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbbfff640 1 --2- 172.21.15.62:0/1909960847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efda40039c0 0x7efda4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbb7fe640 1 --2- 172.21.15.62:0/1909960847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efda4004810 0x7efda4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdc0ab5640 1 -- 172.21.15.62:0/1909960847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efda4004810 msgr2=0x7efda4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdc0ab5640 1 --2- 172.21.15.62:0/1909960847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efda4004810 0x7efda4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdc0ab5640 1 -- 172.21.15.62:0/1909960847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efda40039c0 msgr2=0x7efda4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdc0ab5640 1 --2- 172.21.15.62:0/1909960847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efda40039c0 0x7efda4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdc0ab5640 1 -- 172.21.15.62:0/1909960847 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda4003530 con 0x7efda40052f0 2024-04-18T10:06:18.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.730+0000 7efdbbfff640 1 --2- 172.21.15.62:0/1909960847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efda40039c0 0x7efda4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:18.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdc0ab5640 1 -- 172.21.15.62:0/1909960847 reap_dead start 2024-04-18T10:06:18.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdc0ab5640 1 --2- 172.21.15.62:0/1909960847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efda40052f0 0x7efda400b9d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efdbc13f4d0 tx=0x7efdb401d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1c5578f1edce3ff server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:18.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb400a480 con 0x7efda40052f0 2024-04-18T10:06:18.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdb40ad630 con 0x7efda40052f0 2024-04-18T10:06:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdbaffd640 1 -- 172.21.15.62:0/1909960847 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb40ada20 con 0x7efda40052f0 2024-04-18T10:06:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdc2518640 1 -- 172.21.15.62:0/1909960847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efda40052f0 msgr2=0x7efda400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdc2518640 1 --2- 172.21.15.62:0/1909960847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efda40052f0 0x7efda400b9d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efdbc13f4d0 tx=0x7efdb401d320 comp rx=0 tx=0).stop 2024-04-18T10:06:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdc2518640 1 -- 172.21.15.62:0/1909960847 shutdown_connections 2024-04-18T10:06:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdc2518640 1 --2- 172.21.15.62:0/1909960847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efda40052f0 0x7efda400b9d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdc2518640 1 -- 172.21.15.62:0/1909960847 >> 172.21.15.62:0/1909960847 conn(0x7efdbc0b9690 msgr2=0x7efdbc0b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdc2518640 1 -- 172.21.15.62:0/1909960847 shutdown_connections 2024-04-18T10:06:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdc2518640 1 -- 172.21.15.62:0/1909960847 wait complete. 2024-04-18T10:06:18.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdc2518640 1 Processor -- start 2024-04-18T10:06:18.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.734+0000 7efdc2518640 1 -- start start 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdc2518640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efda40052f0 0x7efdbc141ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdc2518640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc142400 0x7efdbc139e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdc2518640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdbc13a370 0x7efdbc13b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdc2518640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc06d2a0 con 0x7efdbc142400 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdc2518640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7efdbc06e470 con 0x7efdbc13a370 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdc2518640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7efdbc06d420 con 0x7efda40052f0 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdbb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc142400 0x7efdbc139e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdbb7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc142400 0x7efdbc139e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41990/0 (socket says 172.21.15.62:41990) 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdbb7fe640 1 -- 172.21.15.62:0/1771455530 learned_addr learned my addr 172.21.15.62:0/1771455530 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdbbfff640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efda40052f0 0x7efdbc141ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdc0ab5640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdbc13a370 0x7efdbc13b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdbb7fe640 1 -- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efda40052f0 msgr2=0x7efdbc141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdbb7fe640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efda40052f0 0x7efdbc141ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdbb7fe640 1 -- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdbc13a370 msgr2=0x7efdbc13b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdbb7fe640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdbc13a370 0x7efdbc13b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdbb7fe640 1 -- 172.21.15.62:0/1771455530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdbc13de20 con 0x7efdbc142400 2024-04-18T10:06:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdbbfff640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efda40052f0 0x7efdbc141ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdbb7fe640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc142400 0x7efdbc139e30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efdb00105d0 tx=0x7efdb0010a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdb8ff9640 1 -- 172.21.15.62:0/1771455530 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0003600 con 0x7efdbc142400 2024-04-18T10:06:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdb8ff9640 1 -- 172.21.15.62:0/1771455530 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdb00037a0 con 0x7efdbc142400 2024-04-18T10:06:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdc2518640 1 -- 172.21.15.62:0/1771455530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdbc142b70 con 0x7efdbc142400 2024-04-18T10:06:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdb8ff9640 1 -- 172.21.15.62:0/1771455530 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0003b90 con 0x7efdbc142400 2024-04-18T10:06:18.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdc2518640 1 -- 172.21.15.62:0/1771455530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdbc13f980 con 0x7efdbc142400 2024-04-18T10:06:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdc2518640 1 -- 172.21.15.62:0/1771455530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdbc068c00 con 0x7efdbc142400 2024-04-18T10:06:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdb8ff9640 1 -- 172.21.15.62:0/1771455530 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efdb000e5f0 con 0x7efdbc142400 2024-04-18T10:06:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdb8ff9640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efda80683b0 0x7efda806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdb8ff9640 1 -- 172.21.15.62:0/1771455530 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efdb0088ae0 con 0x7efdbc142400 2024-04-18T10:06:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.738+0000 7efdbbfff640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efda80683b0 0x7efda806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:18.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.742+0000 7efdbbfff640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efda80683b0 0x7efda806a870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efdac0099b0 tx=0x7efdac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:18.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.742+0000 7efdb8ff9640 1 -- 172.21.15.62:0/1771455530 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efdb0016030 con 0x7efdbc142400 2024-04-18T10:06:18.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.906+0000 7efdc2518640 1 -- 172.21.15.62:0/1771455530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efdbc0731c0 con 0x7efdbc142400 2024-04-18T10:06:18.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.906+0000 7efdb8ff9640 1 -- 172.21.15.62:0/1771455530 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7efdb0059130 con 0x7efdbc142400 2024-04-18T10:06:18.907 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:18.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.910+0000 7efdc2518640 1 -- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efda80683b0 msgr2=0x7efda806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.910+0000 7efdc2518640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efda80683b0 0x7efda806a870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efdac0099b0 tx=0x7efdac008040 comp rx=0 tx=0).stop 2024-04-18T10:06:18.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.910+0000 7efdc2518640 1 -- 172.21.15.62:0/1771455530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc142400 msgr2=0x7efdbc139e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:18.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.910+0000 7efdc2518640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc142400 0x7efdbc139e30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efdb00105d0 tx=0x7efdb0010a90 comp rx=0 tx=0).stop 2024-04-18T10:06:18.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.910+0000 7efdc2518640 1 -- 172.21.15.62:0/1771455530 shutdown_connections 2024-04-18T10:06:18.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.910+0000 7efdc2518640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efda80683b0 0x7efda806a870 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.910+0000 7efdc2518640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdbc13a370 0x7efdbc13b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.910+0000 7efdc2518640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdbc142400 0x7efdbc139e30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.910+0000 7efdc2518640 1 --2- 172.21.15.62:0/1771455530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efda40052f0 0x7efdbc141ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:18.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.910+0000 7efdc2518640 1 -- 172.21.15.62:0/1771455530 >> 172.21.15.62:0/1771455530 conn(0x7efdbc0b9690 msgr2=0x7efdbc145350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:18.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.910+0000 7efdc2518640 1 -- 172.21.15.62:0/1771455530 shutdown_connections 2024-04-18T10:06:18.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:18.910+0000 7efdc2518640 1 -- 172.21.15.62:0/1771455530 wait complete. 2024-04-18T10:06:18.926 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-18T10:06:19.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.050+0000 7f8c7a2cc640 1 Processor -- start 2024-04-18T10:06:19.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.050+0000 7f8c7a2cc640 1 -- start start 2024-04-18T10:06:19.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c7a2cc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c740b4c90 0x7f8c74145f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c7a2cc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8c7406e490 0x7f8c741464c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c7a2cc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8c7409e090 0x7f8c7409e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c7a2cc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f8c7406d2a0 con 0x7f8c7406e490 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c7a2cc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f8c7409ea60 con 0x7f8c7409e090 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c7a2cc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8c7406d4f0 con 0x7f8c740b4c90 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c737fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8c7406e490 0x7f8c741464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c737fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8c7406e490 0x7f8c741464c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41998/0 (socket says 172.21.15.62:41998) 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c737fe640 1 -- 172.21.15.62:0/3975818406 learned_addr learned my addr 172.21.15.62:0/3975818406 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c73fff640 1 --2- 172.21.15.62:0/3975818406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c740b4c90 0x7f8c74145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c78869640 1 --2- 172.21.15.62:0/3975818406 >> v2:172.21.15.145:3300/0 conn(0x7f8c7409e090 0x7f8c7409e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c737fe640 1 -- 172.21.15.62:0/3975818406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c740b4c90 msgr2=0x7f8c74145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c737fe640 1 --2- 172.21.15.62:0/3975818406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c740b4c90 0x7f8c74145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c737fe640 1 -- 172.21.15.62:0/3975818406 >> v2:172.21.15.145:3300/0 conn(0x7f8c7409e090 msgr2=0x7f8c7409e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c737fe640 1 --2- 172.21.15.62:0/3975818406 >> v2:172.21.15.145:3300/0 conn(0x7f8c7409e090 0x7f8c7409e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c737fe640 1 -- 172.21.15.62:0/3975818406 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7406d7d0 con 0x7f8c7406e490 2024-04-18T10:06:19.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c73fff640 1 --2- 172.21.15.62:0/3975818406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c740b4c90 0x7f8c74145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:19.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c78869640 1 --2- 172.21.15.62:0/3975818406 >> v2:172.21.15.145:3300/0 conn(0x7f8c7409e090 0x7f8c7409e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:19.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c737fe640 1 --2- 172.21.15.62:0/3975818406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7406e490 0x7f8c741464c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8c7409ca80 tx=0x7f8c6801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f43a0ac04f7cf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:19.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c72ffd640 1 -- 172.21.15.62:0/3975818406 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c680b2070 con 0x7f8c7406e490 2024-04-18T10:06:19.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c72ffd640 1 -- 172.21.15.62:0/3975818406 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c680a9070 con 0x7f8c7406e490 2024-04-18T10:06:19.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c72ffd640 1 -- 172.21.15.62:0/3975818406 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c680ad880 con 0x7f8c7406e490 2024-04-18T10:06:19.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/3975818406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7406e490 msgr2=0x7f8c741464c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.054+0000 7f8c7a2cc640 1 --2- 172.21.15.62:0/3975818406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7406e490 0x7f8c741464c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8c7409ca80 tx=0x7f8c6801c490 comp rx=0 tx=0).stop 2024-04-18T10:06:19.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/3975818406 shutdown_connections 2024-04-18T10:06:19.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 --2- 172.21.15.62:0/3975818406 >> v2:172.21.15.145:3300/0 conn(0x7f8c7409e090 0x7f8c7409e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 --2- 172.21.15.62:0/3975818406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7406e490 0x7f8c741464c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 --2- 172.21.15.62:0/3975818406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c740b4c90 0x7f8c74145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/3975818406 >> 172.21.15.62:0/3975818406 conn(0x7f8c740b9690 msgr2=0x7f8c740b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:19.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/3975818406 shutdown_connections 2024-04-18T10:06:19.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/3975818406 wait complete. 2024-04-18T10:06:19.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 Processor -- start 2024-04-18T10:06:19.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 -- start start 2024-04-18T10:06:19.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c7406e490 0x7f8c74148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7409e090 0x7f8c740af650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c740b4c90 0x7f8c741e36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8c74149a30 con 0x7f8c7409e090 2024-04-18T10:06:19.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f8c741498b0 con 0x7f8c7406e490 2024-04-18T10:06:19.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8c74149bb0 con 0x7f8c740b4c90 2024-04-18T10:06:19.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c737fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7409e090 0x7f8c740af650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c737fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7409e090 0x7f8c740af650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42012/0 (socket says 172.21.15.62:42012) 2024-04-18T10:06:19.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c737fe640 1 -- 172.21.15.62:0/2883887760 learned_addr learned my addr 172.21.15.62:0/2883887760 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:19.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c73fff640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c7406e490 0x7f8c74148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c78869640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c740b4c90 0x7f8c741e36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c737fe640 1 -- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c740b4c90 msgr2=0x7f8c741e36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c737fe640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c740b4c90 0x7f8c741e36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c737fe640 1 -- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c7406e490 msgr2=0x7f8c74148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c737fe640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c7406e490 0x7f8c74148790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c737fe640 1 -- 172.21.15.62:0/2883887760 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c7414f3f0 con 0x7f8c7409e090 2024-04-18T10:06:19.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c737fe640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7409e090 0x7f8c740af650 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8c6801ab40 tx=0x7f8c68019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:19.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c70ff9640 1 -- 172.21.15.62:0/2883887760 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c680b2040 con 0x7f8c7409e090 2024-04-18T10:06:19.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c70ff9640 1 -- 172.21.15.62:0/2883887760 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c680b6620 con 0x7f8c7409e090 2024-04-18T10:06:19.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/2883887760 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c740afe10 con 0x7f8c7409e090 2024-04-18T10:06:19.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c70ff9640 1 -- 172.21.15.62:0/2883887760 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c680b6a10 con 0x7f8c7409e090 2024-04-18T10:06:19.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/2883887760 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c74148e20 con 0x7f8c7409e090 2024-04-18T10:06:19.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.058+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/2883887760 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c74068c00 con 0x7f8c7409e090 2024-04-18T10:06:19.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.062+0000 7f8c70ff9640 1 -- 172.21.15.62:0/2883887760 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8c680ab070 con 0x7f8c7409e090 2024-04-18T10:06:19.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.062+0000 7f8c70ff9640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8c60068380 0x7f8c6006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.062+0000 7f8c70ff9640 1 -- 172.21.15.62:0/2883887760 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8c68120580 con 0x7f8c7409e090 2024-04-18T10:06:19.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.062+0000 7f8c73fff640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8c60068380 0x7f8c6006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.062+0000 7f8c73fff640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8c60068380 0x7f8c6006a840 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c007510 tx=0x7f8c6c0074a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:19.065 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.066+0000 7f8c70ff9640 1 -- 172.21.15.62:0/2883887760 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8c68016110 con 0x7f8c7409e090 2024-04-18T10:06:19.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.226+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/2883887760 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8c740b44c0 con 0x7f8c7409e090 2024-04-18T10:06:19.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c70ff9640 1 -- 172.21.15.62:0/2883887760 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f8c680f0cf0 con 0x7f8c7409e090 2024-04-18T10:06:19.227 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-18T10:06:19.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8c60068380 msgr2=0x7f8c6006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c7a2cc640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8c60068380 0x7f8c6006a840 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c007510 tx=0x7f8c6c0074a0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/2883887760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7409e090 msgr2=0x7f8c740af650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c7a2cc640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7409e090 0x7f8c740af650 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8c6801ab40 tx=0x7f8c68019230 comp rx=0 tx=0).stop 2024-04-18T10:06:19.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/2883887760 shutdown_connections 2024-04-18T10:06:19.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c7a2cc640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8c60068380 0x7f8c6006a840 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c7a2cc640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8c740b4c90 0x7f8c741e36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c7a2cc640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c7409e090 0x7f8c740af650 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c7a2cc640 1 --2- 172.21.15.62:0/2883887760 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8c7406e490 0x7f8c74148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/2883887760 >> 172.21.15.62:0/2883887760 conn(0x7f8c740b9690 msgr2=0x7f8c7409cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:19.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/2883887760 shutdown_connections 2024-04-18T10:06:19.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.230+0000 7f8c7a2cc640 1 -- 172.21.15.62:0/2883887760 wait complete. 2024-04-18T10:06:19.248 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-18T10:06:19.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdee7d640 1 Processor -- start 2024-04-18T10:06:19.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdee7d640 1 -- start start 2024-04-18T10:06:19.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdee7d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd806dd90 0x7f3fd80b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdee7d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3fd8144950 0x7f3fd814eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdee7d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fd806e170 con 0x7f3fd8147840 2024-04-18T10:06:19.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdee7d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f3fd80af620 con 0x7f3fd8144950 2024-04-18T10:06:19.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdee7d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3fd80af850 con 0x7f3fd806dd90 2024-04-18T10:06:19.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdd41a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3fd8144950 0x7f3fd814eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fd7fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3fd8147840 0x7f3fd8147c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37202/0 (socket says 172.21.15.62:37202) 2024-04-18T10:06:19.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fd7fff640 1 -- 172.21.15.62:0/650986033 learned_addr learned my addr 172.21.15.62:0/650986033 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:19.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdcc19640 1 --2- 172.21.15.62:0/650986033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd806dd90 0x7f3fd80b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdd41a640 1 -- 172.21.15.62:0/650986033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd806dd90 msgr2=0x7f3fd80b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:19.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdd41a640 1 --2- 172.21.15.62:0/650986033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd806dd90 0x7f3fd80b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdd41a640 1 -- 172.21.15.62:0/650986033 >> v1:172.21.15.62:6789/0 conn(0x7f3fd8147840 legacy=0x7f3fd8147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fdd41a640 1 -- 172.21.15.62:0/650986033 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd80afb30 con 0x7f3fd8144950 2024-04-18T10:06:19.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.374+0000 7f3fd77fe640 1 -- 172.21.15.62:0/650986033 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 133434158 0 0) 0x7f3fd806e170 con 0x7f3fd8147840 2024-04-18T10:06:19.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fdd41a640 1 --2- 172.21.15.62:0/650986033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fd8144950 0x7f3fd814eb70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009d90 tx=0x7f3fd001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5176bc454eab56c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:19.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fd77fe640 1 -- 172.21.15.62:0/650986033 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00b2070 con 0x7f3fd8144950 2024-04-18T10:06:19.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fd77fe640 1 -- 172.21.15.62:0/650986033 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fd00a57a0 con 0x7f3fd8144950 2024-04-18T10:06:19.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fd77fe640 1 -- 172.21.15.62:0/650986033 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00a5b70 con 0x7f3fd8144950 2024-04-18T10:06:19.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fdee7d640 1 -- 172.21.15.62:0/650986033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fd8144950 msgr2=0x7f3fd814eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fdee7d640 1 --2- 172.21.15.62:0/650986033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fd8144950 0x7f3fd814eb70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009d90 tx=0x7f3fd001c8e0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fdee7d640 1 -- 172.21.15.62:0/650986033 shutdown_connections 2024-04-18T10:06:19.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fdee7d640 1 --2- 172.21.15.62:0/650986033 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fd8144950 0x7f3fd814eb70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fdee7d640 1 --2- 172.21.15.62:0/650986033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd806dd90 0x7f3fd80b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fdee7d640 1 -- 172.21.15.62:0/650986033 >> 172.21.15.62:0/650986033 conn(0x7f3fd80b9690 msgr2=0x7f3fd80b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:19.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fdee7d640 1 -- 172.21.15.62:0/650986033 shutdown_connections 2024-04-18T10:06:19.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fdee7d640 1 -- 172.21.15.62:0/650986033 wait complete. 2024-04-18T10:06:19.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fdee7d640 1 Processor -- start 2024-04-18T10:06:19.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.378+0000 7f3fdee7d640 1 -- start start 2024-04-18T10:06:19.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdee7d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd806dd90 0x7f3fd81471b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdee7d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fd8144950 0x7f3fd8145ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdee7d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fd8147840 0x7f3fd81460e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdee7d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd806d420 con 0x7f3fd8147840 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdee7d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd806d2a0 con 0x7f3fd8144950 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdee7d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3fd806d5a0 con 0x7f3fd806dd90 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdd41a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fd8147840 0x7f3fd81460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdd41a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fd8147840 0x7f3fd81460e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42026/0 (socket says 172.21.15.62:42026) 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdd41a640 1 -- 172.21.15.62:0/3502453536 learned_addr learned my addr 172.21.15.62:0/3502453536 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fd7fff640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fd8144950 0x7f3fd8145ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdcc19640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd806dd90 0x7f3fd81471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdd41a640 1 -- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd806dd90 msgr2=0x7f3fd81471b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdd41a640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd806dd90 0x7f3fd81471b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdd41a640 1 -- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fd8144950 msgr2=0x7f3fd8145ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdd41a640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fd8144950 0x7f3fd8145ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdd41a640 1 -- 172.21.15.62:0/3502453536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd813b8b0 con 0x7f3fd8147840 2024-04-18T10:06:19.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fd7fff640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fd8144950 0x7f3fd8145ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:19.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdcc19640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd806dd90 0x7f3fd81471b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:19.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdd41a640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fd8147840 0x7f3fd81460e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3fd001af90 tx=0x7f3fd0019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:19.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fd57fa640 1 -- 172.21.15.62:0/3502453536 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00b2050 con 0x7f3fd8147840 2024-04-18T10:06:19.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fd57fa640 1 -- 172.21.15.62:0/3502453536 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fd00be070 con 0x7f3fd8147840 2024-04-18T10:06:19.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdee7d640 1 -- 172.21.15.62:0/3502453536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fd813f0c0 con 0x7f3fd8147840 2024-04-18T10:06:19.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fd57fa640 1 -- 172.21.15.62:0/3502453536 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00ad850 con 0x7f3fd8147840 2024-04-18T10:06:19.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdee7d640 1 -- 172.21.15.62:0/3502453536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fd813fa10 con 0x7f3fd8147840 2024-04-18T10:06:19.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdee7d640 1 -- 172.21.15.62:0/3502453536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fd8068c00 con 0x7f3fd8147840 2024-04-18T10:06:19.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fd57fa640 1 -- 172.21.15.62:0/3502453536 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3fd00ad9f0 con 0x7f3fd8147840 2024-04-18T10:06:19.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fd57fa640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3fbc068380 0x7f3fbc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fd57fa640 1 -- 172.21.15.62:0/3502453536 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3fd0121bf0 con 0x7f3fd8147840 2024-04-18T10:06:19.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.382+0000 7f3fdcc19640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3fbc068380 0x7f3fbc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.386+0000 7f3fdcc19640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3fbc068380 0x7f3fbc06a840 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc009840 tx=0x7f3fcc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:19.387 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.386+0000 7f3fd57fa640 1 -- 172.21.15.62:0/3502453536 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3fd0016560 con 0x7f3fd8147840 2024-04-18T10:06:19.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 -- 172.21.15.62:0/3502453536 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3fd809bcd0 con 0x7f3fd8147840 2024-04-18T10:06:19.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fd57fa640 1 -- 172.21.15.62:0/3502453536 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f3fd00f2360 con 0x7f3fd8147840 2024-04-18T10:06:19.552 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-04-18T10:06:19.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 -- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3fbc068380 msgr2=0x7f3fbc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3fbc068380 0x7f3fbc06a840 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc009840 tx=0x7f3fcc008040 comp rx=0 tx=0).stop 2024-04-18T10:06:19.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 -- 172.21.15.62:0/3502453536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fd8147840 msgr2=0x7f3fd81460e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fd8147840 0x7f3fd81460e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3fd001af90 tx=0x7f3fd0019680 comp rx=0 tx=0).stop 2024-04-18T10:06:19.554 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 -- 172.21.15.62:0/3502453536 shutdown_connections 2024-04-18T10:06:19.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3fbc068380 0x7f3fbc06a840 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fd8147840 0x7f3fd81460e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fd8144950 0x7f3fd8145ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 --2- 172.21.15.62:0/3502453536 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fd806dd90 0x7f3fd81471b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 -- 172.21.15.62:0/3502453536 >> 172.21.15.62:0/3502453536 conn(0x7f3fd80b9690 msgr2=0x7f3fd809aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:19.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 -- 172.21.15.62:0/3502453536 shutdown_connections 2024-04-18T10:06:19.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.554+0000 7f3fdee7d640 1 -- 172.21.15.62:0/3502453536 wait complete. 2024-04-18T10:06:19.616 INFO:tasks.cephfs_test_runner:Starting test: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) Test that when cd is issued with an argument, CWD is changed 2024-04-18T10:06:19.616 INFO:tasks.cephfs_test_runner:test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-04-18T10:06:19.616 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:19.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.730+0000 7f90999fa640 1 Processor -- start 2024-04-18T10:06:19.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.730+0000 7f90999fa640 1 -- start start 2024-04-18T10:06:19.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f90999fa640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f90940b9d90 0x7f909406d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f90999fa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9094145f20 0x7f90941504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f90999fa640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90940ba170 con 0x7f909409be30 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f90999fa640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f909409cd30 con 0x7f90940b9d90 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f90999fa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f909414a060 con 0x7f9094145f20 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f90927fc640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f909409be30 0x7f909406d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37218/0 (socket says 172.21.15.62:37218) 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f90927fc640 1 -- 172.21.15.62:0/3299706847 learned_addr learned my addr 172.21.15.62:0/3299706847 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f9092ffd640 1 --2- 172.21.15.62:0/3299706847 >> v2:172.21.15.62:3300/0 conn(0x7f90940b9d90 0x7f909406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f90937fe640 1 --2- 172.21.15.62:0/3299706847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9094145f20 0x7f90941504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f9091ffb640 1 -- 172.21.15.62:0/3299706847 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3293106774 0 0) 0x7f90940ba170 con 0x7f909409be30 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f9091ffb640 1 -- 172.21.15.62:0/3299706847 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9080003530 con 0x7f909409be30 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f9092ffd640 1 -- 172.21.15.62:0/3299706847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9094145f20 msgr2=0x7f90941504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f9092ffd640 1 --2- 172.21.15.62:0/3299706847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9094145f20 0x7f90941504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f9092ffd640 1 -- 172.21.15.62:0/3299706847 >> v1:172.21.15.62:6789/0 conn(0x7f909409be30 legacy=0x7f909406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f9092ffd640 1 -- 172.21.15.62:0/3299706847 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9094149860 con 0x7f90940b9d90 2024-04-18T10:06:19.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f9092ffd640 1 --2- 172.21.15.62:0/3299706847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90940b9d90 0x7f909406d210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f908c0097c0 tx=0x7f908c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d72ae6d72a058f server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:19.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f9091ffb640 1 -- 172.21.15.62:0/3299706847 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c0b0070 con 0x7f90940b9d90 2024-04-18T10:06:19.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f9091ffb640 1 -- 172.21.15.62:0/3299706847 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f908c0ab600 con 0x7f90940b9d90 2024-04-18T10:06:19.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.734+0000 7f9091ffb640 1 -- 172.21.15.62:0/3299706847 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908c0ab9d0 con 0x7f90940b9d90 2024-04-18T10:06:19.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 -- 172.21.15.62:0/3299706847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90940b9d90 msgr2=0x7f909406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 --2- 172.21.15.62:0/3299706847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90940b9d90 0x7f909406d210 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f908c0097c0 tx=0x7f908c01c380 comp rx=0 tx=0).stop 2024-04-18T10:06:19.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 -- 172.21.15.62:0/3299706847 shutdown_connections 2024-04-18T10:06:19.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 --2- 172.21.15.62:0/3299706847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9094145f20 0x7f90941504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 --2- 172.21.15.62:0/3299706847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f90940b9d90 0x7f909406d210 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 -- 172.21.15.62:0/3299706847 >> 172.21.15.62:0/3299706847 conn(0x7f909406c0d0 msgr2=0x7f90940b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:19.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 -- 172.21.15.62:0/3299706847 shutdown_connections 2024-04-18T10:06:19.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 -- 172.21.15.62:0/3299706847 wait complete. 2024-04-18T10:06:19.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 Processor -- start 2024-04-18T10:06:19.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 -- start start 2024-04-18T10:06:19.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f909409be30 0x7f909413cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90940b9d90 0x7f90941e0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9094145f20 0x7f90941e0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f90940b3fa0 con 0x7f9094145f20 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f90940b3e20 con 0x7f909409be30 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f909406de60 con 0x7f90940b9d90 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90937fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9094145f20 0x7f90941e0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90937fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9094145f20 0x7f90941e0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42052/0 (socket says 172.21.15.62:42052) 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90937fe640 1 -- 172.21.15.62:0/4219662920 learned_addr learned my addr 172.21.15.62:0/4219662920 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f9092ffd640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f909409be30 0x7f909413cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90937fe640 1 -- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90940b9d90 msgr2=0x7f90941e0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90927fc640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90940b9d90 0x7f90941e0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90937fe640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90940b9d90 0x7f90941e0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90937fe640 1 -- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f909409be30 msgr2=0x7f909413cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90937fe640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f909409be30 0x7f909413cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90937fe640 1 -- 172.21.15.62:0/4219662920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f909413c260 con 0x7f9094145f20 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90937fe640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9094145f20 0x7f90941e0c40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f908400d560 tx=0x7f908400da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f908bfff640 1 -- 172.21.15.62:0/4219662920 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90840125f0 con 0x7f9094145f20 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f908bfff640 1 -- 172.21.15.62:0/4219662920 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9084019070 con 0x7f9094145f20 2024-04-18T10:06:19.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 -- 172.21.15.62:0/4219662920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f909409b160 con 0x7f9094145f20 2024-04-18T10:06:19.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f908bfff640 1 -- 172.21.15.62:0/4219662920 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90840129f0 con 0x7f9094145f20 2024-04-18T10:06:19.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 -- 172.21.15.62:0/4219662920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90941411c0 con 0x7f9094145f20 2024-04-18T10:06:19.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.738+0000 7f90999fa640 1 -- 172.21.15.62:0/4219662920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9094068b90 con 0x7f9094145f20 2024-04-18T10:06:19.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.742+0000 7f908bfff640 1 -- 172.21.15.62:0/4219662920 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9084012b90 con 0x7f9094145f20 2024-04-18T10:06:19.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.742+0000 7f908bfff640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9074068380 0x7f907406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:19.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.742+0000 7f908bfff640 1 -- 172.21.15.62:0/4219662920 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9084086cc0 con 0x7f9094145f20 2024-04-18T10:06:19.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.742+0000 7f9092ffd640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9074068380 0x7f907406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:19.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.742+0000 7f9092ffd640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9074068380 0x7f907406a840 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f908c0ad000 tx=0x7f908c0a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:19.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.746+0000 7f908bfff640 1 -- 172.21.15.62:0/4219662920 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9084057310 con 0x7f9094145f20 2024-04-18T10:06:19.902 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.902+0000 7f90999fa640 1 -- 172.21.15.62:0/4219662920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9094143200 con 0x7f9094145f20 2024-04-18T10:06:19.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f908bfff640 1 -- 172.21.15.62:0/4219662920 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f9084056c10 con 0x7f9094145f20 2024-04-18T10:06:19.904 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:19.904 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:19.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f90999fa640 1 -- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9074068380 msgr2=0x7f907406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f90999fa640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9074068380 0x7f907406a840 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f908c0ad000 tx=0x7f908c0a6000 comp rx=0 tx=0).stop 2024-04-18T10:06:19.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f90999fa640 1 -- 172.21.15.62:0/4219662920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9094145f20 msgr2=0x7f90941e0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:19.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f90999fa640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9094145f20 0x7f90941e0c40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f908400d560 tx=0x7f908400da20 comp rx=0 tx=0).stop 2024-04-18T10:06:19.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f90999fa640 1 -- 172.21.15.62:0/4219662920 shutdown_connections 2024-04-18T10:06:19.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f90999fa640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9074068380 0x7f907406a840 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f90999fa640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9094145f20 0x7f90941e0c40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f90999fa640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90940b9d90 0x7f90941e0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f90999fa640 1 --2- 172.21.15.62:0/4219662920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f909409be30 0x7f909413cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:19.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f90999fa640 1 -- 172.21.15.62:0/4219662920 >> 172.21.15.62:0/4219662920 conn(0x7f909406c0d0 msgr2=0x7f90940aefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:19.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f90999fa640 1 -- 172.21.15.62:0/4219662920 shutdown_connections 2024-04-18T10:06:19.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:19.906+0000 7f90999fa640 1 -- 172.21.15.62:0/4219662920 wait complete. 2024-04-18T10:06:19.922 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-18T10:05:56.782649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-18T10:05:57.541623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:19.923 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:20.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f3431d31640 1 Processor -- start 2024-04-18T10:06:20.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f3431d31640 1 -- start start 2024-04-18T10:06:20.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f3431d31640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f342c0bbfb0 0x7f342c0b7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f3431d31640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f342c06cbe0 0x7f342c141950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f3431d31640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f342c0bc390 con 0x7f342c14b250 2024-04-18T10:06:20.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f3431d31640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f342c06e640 con 0x7f342c0bbfb0 2024-04-18T10:06:20.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f3431d31640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f342c06e920 con 0x7f342c06cbe0 2024-04-18T10:06:20.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f342b7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f342c0bbfb0 0x7f342c0b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f342b7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f342c0bbfb0 0x7f342c0b7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42062/0 (socket says 172.21.15.62:42062) 2024-04-18T10:06:20.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f342b7fe640 1 -- 172.21.15.62:0/3997343492 learned_addr learned my addr 172.21.15.62:0/3997343492 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:20.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f342b7fe640 1 -- 172.21.15.62:0/3997343492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f342c06cbe0 msgr2=0x7f342c141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f342affd640 1 --2- 172.21.15.62:0/3997343492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f342c06cbe0 0x7f342c141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f342b7fe640 1 --2- 172.21.15.62:0/3997343492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f342c06cbe0 0x7f342c141950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f342b7fe640 1 -- 172.21.15.62:0/3997343492 >> v1:172.21.15.145:6789/0 conn(0x7f342c14b250 legacy=0x7f342c072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f342b7fe640 1 -- 172.21.15.62:0/3997343492 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342c06e2d0 con 0x7f342c0bbfb0 2024-04-18T10:06:20.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.046+0000 7f342affd640 1 --2- 172.21.15.62:0/3997343492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f342c06cbe0 0x7f342c141950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:20.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342b7fe640 1 --2- 172.21.15.62:0/3997343492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f342c0bbfb0 0x7f342c0b7210 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f341c009870 tx=0x7f341c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804c0c26803585c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:20.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342a7fc640 1 -- 172.21.15.62:0/3997343492 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c0b0070 con 0x7f342c0bbfb0 2024-04-18T10:06:20.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342a7fc640 1 -- 172.21.15.62:0/3997343492 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f341c0ab600 con 0x7f342c0bbfb0 2024-04-18T10:06:20.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342a7fc640 1 -- 172.21.15.62:0/3997343492 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341c0ab9d0 con 0x7f342c0bbfb0 2024-04-18T10:06:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 -- 172.21.15.62:0/3997343492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f342c0bbfb0 msgr2=0x7f342c0b7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 --2- 172.21.15.62:0/3997343492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f342c0bbfb0 0x7f342c0b7210 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f341c009870 tx=0x7f341c01c520 comp rx=0 tx=0).stop 2024-04-18T10:06:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 -- 172.21.15.62:0/3997343492 shutdown_connections 2024-04-18T10:06:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 --2- 172.21.15.62:0/3997343492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f342c06cbe0 0x7f342c141950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 --2- 172.21.15.62:0/3997343492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f342c0bbfb0 0x7f342c0b7210 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 -- 172.21.15.62:0/3997343492 >> 172.21.15.62:0/3997343492 conn(0x7f342c067e40 msgr2=0x7f342c0bb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 -- 172.21.15.62:0/3997343492 shutdown_connections 2024-04-18T10:06:20.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 -- 172.21.15.62:0/3997343492 wait complete. 2024-04-18T10:06:20.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 Processor -- start 2024-04-18T10:06:20.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 -- start start 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f342c06cbe0 0x7f342c14a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f342c0bbfb0 0x7f342c1394a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f342c14b250 0x7f342c1399e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f342c155780 con 0x7f342c0bbfb0 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f342c155600 con 0x7f342c14b250 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f342c155900 con 0x7f342c06cbe0 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f342c0bbfb0 0x7f342c1394a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f342c0bbfb0 0x7f342c1394a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42078/0 (socket says 172.21.15.62:42078) 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342affd640 1 -- 172.21.15.62:0/230197514 learned_addr learned my addr 172.21.15.62:0/230197514 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342b7fe640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f342c06cbe0 0x7f342c14a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342bfff640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f342c14b250 0x7f342c1399e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342affd640 1 -- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f342c06cbe0 msgr2=0x7f342c14a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342affd640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f342c06cbe0 0x7f342c14a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342affd640 1 -- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f342c14b250 msgr2=0x7f342c1399e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342affd640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f342c14b250 0x7f342c1399e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342affd640 1 -- 172.21.15.62:0/230197514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342c13b650 con 0x7f342c0bbfb0 2024-04-18T10:06:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f342affd640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f342c0bbfb0 0x7f342c1394a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f342000c7c0 tx=0x7f342000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3413fff640 1 -- 172.21.15.62:0/230197514 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3420007a30 con 0x7f342c0bbfb0 2024-04-18T10:06:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3413fff640 1 -- 172.21.15.62:0/230197514 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3420007bd0 con 0x7f342c0bbfb0 2024-04-18T10:06:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 -- 172.21.15.62:0/230197514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f342c13adb0 con 0x7f342c0bbfb0 2024-04-18T10:06:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3413fff640 1 -- 172.21.15.62:0/230197514 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34200113f0 con 0x7f342c0bbfb0 2024-04-18T10:06:20.052 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.050+0000 7f3431d31640 1 -- 172.21.15.62:0/230197514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f342c145400 con 0x7f342c0bbfb0 2024-04-18T10:06:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.054+0000 7f3431d31640 1 -- 172.21.15.62:0/230197514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f342c068c00 con 0x7f342c0bbfb0 2024-04-18T10:06:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.054+0000 7f3413fff640 1 -- 172.21.15.62:0/230197514 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3420011590 con 0x7f342c0bbfb0 2024-04-18T10:06:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.054+0000 7f3413fff640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3418068380 0x7f341806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.054+0000 7f3413fff640 1 -- 172.21.15.62:0/230197514 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3420086ba0 con 0x7f342c0bbfb0 2024-04-18T10:06:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.054+0000 7f342b7fe640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3418068380 0x7f341806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.054+0000 7f342b7fe640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3418068380 0x7f341806a840 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f341c0ad000 tx=0x7f341c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:20.057 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.058+0000 7f3413fff640 1 -- 172.21.15.62:0/230197514 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3420015020 con 0x7f342c0bbfb0 2024-04-18T10:06:20.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.214+0000 7f3431d31640 1 -- 172.21.15.62:0/230197514 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f342c145660 con 0x7f342c0bbfb0 2024-04-18T10:06:20.213 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.214+0000 7f3413fff640 1 -- 172.21.15.62:0/230197514 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f3420057150 con 0x7f342c0bbfb0 2024-04-18T10:06:20.214 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:20.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.218+0000 7f3431d31640 1 -- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3418068380 msgr2=0x7f341806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.218+0000 7f3431d31640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3418068380 0x7f341806a840 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f341c0ad000 tx=0x7f341c008040 comp rx=0 tx=0).stop 2024-04-18T10:06:20.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.218+0000 7f3431d31640 1 -- 172.21.15.62:0/230197514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f342c0bbfb0 msgr2=0x7f342c1394a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.218+0000 7f3431d31640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f342c0bbfb0 0x7f342c1394a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f342000c7c0 tx=0x7f342000cc80 comp rx=0 tx=0).stop 2024-04-18T10:06:20.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.218+0000 7f3431d31640 1 -- 172.21.15.62:0/230197514 shutdown_connections 2024-04-18T10:06:20.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.218+0000 7f3431d31640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3418068380 0x7f341806a840 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.218+0000 7f3431d31640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f342c14b250 0x7f342c1399e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.218+0000 7f3431d31640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f342c0bbfb0 0x7f342c1394a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.218+0000 7f3431d31640 1 --2- 172.21.15.62:0/230197514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f342c06cbe0 0x7f342c14a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.218+0000 7f3431d31640 1 -- 172.21.15.62:0/230197514 >> 172.21.15.62:0/230197514 conn(0x7f342c067e40 msgr2=0x7f342c0bb620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:20.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.218+0000 7f3431d31640 1 -- 172.21.15.62:0/230197514 shutdown_connections 2024-04-18T10:06:20.217 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.218+0000 7f3431d31640 1 -- 172.21.15.62:0/230197514 wait complete. 2024-04-18T10:06:20.231 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-18T10:06:20.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.346+0000 7f4116ffe640 1 Processor -- start 2024-04-18T10:06:20.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.346+0000 7f4116ffe640 1 -- start start 2024-04-18T10:06:20.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f4116ffe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f411006dd90 0x7f41100b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f4116ffe640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f411006e170 con 0x7f4110147840 2024-04-18T10:06:20.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f4116ffe640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41100af850 con 0x7f4110144950 2024-04-18T10:06:20.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f4116ffe640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f41100af620 con 0x7f411006dd90 2024-04-18T10:06:20.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f410ffff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4110147840 0x7f4110147c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37226/0 (socket says 172.21.15.62:37226) 2024-04-18T10:06:20.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f410ffff640 1 -- 172.21.15.62:0/3745643042 learned_addr learned my addr 172.21.15.62:0/3745643042 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:20.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f4114d9a640 1 --2- 172.21.15.62:0/3745643042 >> v2:172.21.15.145:3300/0 conn(0x7f411006dd90 0x7f41100b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f410f7fe640 1 -- 172.21.15.62:0/3745643042 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1593972299 0 0) 0x7f411006e170 con 0x7f4110147840 2024-04-18T10:06:20.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f410f7fe640 1 -- 172.21.15.62:0/3745643042 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4100003530 con 0x7f4110147840 2024-04-18T10:06:20.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f4114d9a640 1 -- 172.21.15.62:0/3745643042 >> v1:172.21.15.145:6789/0 conn(0x7f4110144950 legacy=0x7f411014eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f410f7fe640 1 -- 172.21.15.62:0/3745643042 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4090972185 0 0) 0x7f4100003530 con 0x7f4110147840 2024-04-18T10:06:20.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f4114d9a640 1 -- 172.21.15.62:0/3745643042 >> v1:172.21.15.62:6789/0 conn(0x7f4110147840 legacy=0x7f4110147c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f4114d9a640 1 -- 172.21.15.62:0/3745643042 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41100afb30 con 0x7f411006dd90 2024-04-18T10:06:20.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f4114d9a640 1 --2- 172.21.15.62:0/3745643042 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f411006dd90 0x7f41100b43a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0097c0 tx=0x7f40fc01d440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecbc05f7256ae35d server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:20.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f410f7fe640 1 -- 172.21.15.62:0/3745643042 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc0b1070 con 0x7f411006dd90 2024-04-18T10:06:20.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f410f7fe640 1 -- 172.21.15.62:0/3745643042 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40fc0ac400 con 0x7f411006dd90 2024-04-18T10:06:20.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f410f7fe640 1 -- 172.21.15.62:0/3745643042 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4104001680 con 0x7f411006dd90 2024-04-18T10:06:20.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f4116ffe640 1 -- 172.21.15.62:0/3745643042 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f411006dd90 msgr2=0x7f41100b43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.350+0000 7f4116ffe640 1 --2- 172.21.15.62:0/3745643042 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f411006dd90 0x7f41100b43a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f40fc0097c0 tx=0x7f40fc01d440 comp rx=0 tx=0).stop 2024-04-18T10:06:20.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 -- 172.21.15.62:0/3745643042 shutdown_connections 2024-04-18T10:06:20.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 --2- 172.21.15.62:0/3745643042 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f411006dd90 0x7f41100b43a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 -- 172.21.15.62:0/3745643042 >> 172.21.15.62:0/3745643042 conn(0x7f41100b9690 msgr2=0x7f41100b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:20.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 -- 172.21.15.62:0/3745643042 shutdown_connections 2024-04-18T10:06:20.352 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 -- 172.21.15.62:0/3745643042 wait complete. 2024-04-18T10:06:20.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 Processor -- start 2024-04-18T10:06:20.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 -- start start 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f411006dd90 0x7f41101e4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4110144950 0x7f41101e0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4110147840 0x7f41101e1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4110152420 con 0x7f411006dd90 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f41101522a0 con 0x7f4110144950 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f41101525a0 con 0x7f4110147840 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4114d9a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f411006dd90 0x7f41101e4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4114d9a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f411006dd90 0x7f41101e4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42084/0 (socket says 172.21.15.62:42084) 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4114d9a640 1 -- 172.21.15.62:0/2815734195 learned_addr learned my addr 172.21.15.62:0/2815734195 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4114d9a640 1 -- 172.21.15.62:0/2815734195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4110147840 msgr2=0x7f41101e1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4114d9a640 1 --2- 172.21.15.62:0/2815734195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4110147840 0x7f41101e1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4114d9a640 1 -- 172.21.15.62:0/2815734195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4110144950 msgr2=0x7f41101e0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4114d9a640 1 --2- 172.21.15.62:0/2815734195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4110144950 0x7f41101e0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4114d9a640 1 -- 172.21.15.62:0/2815734195 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411014d000 con 0x7f411006dd90 2024-04-18T10:06:20.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4114d9a640 1 --2- 172.21.15.62:0/2815734195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f411006dd90 0x7f41101e4250 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f40fc01a050 tx=0x7f40fc01a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:20.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f410d7fa640 1 -- 172.21.15.62:0/2815734195 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc0b9020 con 0x7f411006dd90 2024-04-18T10:06:20.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f410d7fa640 1 -- 172.21.15.62:0/2815734195 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40fc0038e0 con 0x7f411006dd90 2024-04-18T10:06:20.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 -- 172.21.15.62:0/2815734195 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41101e5130 con 0x7f411006dd90 2024-04-18T10:06:20.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f410d7fa640 1 -- 172.21.15.62:0/2815734195 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40fc0b1040 con 0x7f411006dd90 2024-04-18T10:06:20.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 -- 172.21.15.62:0/2815734195 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4110145430 con 0x7f411006dd90 2024-04-18T10:06:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f4116ffe640 1 -- 172.21.15.62:0/2815734195 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41101e4e10 con 0x7f411006dd90 2024-04-18T10:06:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.354+0000 7f410d7fa640 1 -- 172.21.15.62:0/2815734195 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f40fc0bc020 con 0x7f411006dd90 2024-04-18T10:06:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.358+0000 7f410d7fa640 1 --2- 172.21.15.62:0/2815734195 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f40f4068380 0x7f40f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.358+0000 7f410d7fa640 1 -- 172.21.15.62:0/2815734195 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f40fc1205a0 con 0x7f411006dd90 2024-04-18T10:06:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.358+0000 7f410ffff640 1 --2- 172.21.15.62:0/2815734195 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f40f4068380 0x7f40f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.358+0000 7f410ffff640 1 --2- 172.21.15.62:0/2815734195 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f40f4068380 0x7f40f406a840 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4104000c10 tx=0x7f41040011b0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:20.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.362+0000 7f410d7fa640 1 -- 172.21.15.62:0/2815734195 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f41101e4e10 con 0x7f411006dd90 2024-04-18T10:06:20.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 -- 172.21.15.62:0/2815734195 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4110068c00 con 0x7f411006dd90 2024-04-18T10:06:20.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f410d7fa640 1 -- 172.21.15.62:0/2815734195 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7f40fc0c3030 con 0x7f411006dd90 2024-04-18T10:06:20.520 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-18T10:06:20.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 -- 172.21.15.62:0/2815734195 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f40f4068380 msgr2=0x7f40f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 --2- 172.21.15.62:0/2815734195 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f40f4068380 0x7f40f406a840 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4104000c10 tx=0x7f41040011b0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 -- 172.21.15.62:0/2815734195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f411006dd90 msgr2=0x7f41101e4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 --2- 172.21.15.62:0/2815734195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f411006dd90 0x7f41101e4250 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f40fc01a050 tx=0x7f40fc01a1e0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 -- 172.21.15.62:0/2815734195 shutdown_connections 2024-04-18T10:06:20.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 --2- 172.21.15.62:0/2815734195 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f40f4068380 0x7f40f406a840 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 --2- 172.21.15.62:0/2815734195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4110147840 0x7f41101e1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 --2- 172.21.15.62:0/2815734195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4110144950 0x7f41101e0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 --2- 172.21.15.62:0/2815734195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f411006dd90 0x7f41101e4250 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 -- 172.21.15.62:0/2815734195 >> 172.21.15.62:0/2815734195 conn(0x7f41100b9690 msgr2=0x7f41100b66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:20.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 -- 172.21.15.62:0/2815734195 shutdown_connections 2024-04-18T10:06:20.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.522+0000 7f4116ffe640 1 -- 172.21.15.62:0/2815734195 wait complete. 2024-04-18T10:06:20.538 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-18T10:06:20.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.666+0000 7fef9239c640 1 Processor -- start 2024-04-18T10:06:20.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.666+0000 7fef9239c640 1 -- start start 2024-04-18T10:06:20.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.666+0000 7fef9239c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fef8c14b190 0x7fef8c147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.666+0000 7fef9239c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef8c14b620 con 0x7fef8c09b9d0 2024-04-18T10:06:20.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.666+0000 7fef9239c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef8c14b900 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.666+0000 7fef9239c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fef8c0b4290 con 0x7fef8c14b190 2024-04-18T10:06:20.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef83fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fef8c14b190 0x7fef8c147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef90939640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fef8c09b9d0 0x7fef8c152520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:37234/0 (socket says 172.21.15.62:37234) 2024-04-18T10:06:20.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef90939640 1 -- 172.21.15.62:0/339021956 learned_addr learned my addr 172.21.15.62:0/339021956 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:20.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef83fff640 1 --2- 172.21.15.62:0/339021956 >> v2:172.21.15.62:3300/0 conn(0x7fef8c14b190 0x7fef8c147ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42092/0 (socket says 172.21.15.62:42092) 2024-04-18T10:06:20.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef83fff640 1 -- 172.21.15.62:0/339021956 >> v1:172.21.15.145:6789/0 conn(0x7fef8c0b3eb0 legacy=0x7fef8c0bdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:20.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef83fff640 1 -- 172.21.15.62:0/339021956 >> v1:172.21.15.62:6789/0 conn(0x7fef8c09b9d0 legacy=0x7fef8c152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef83fff640 1 -- 172.21.15.62:0/339021956 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8c0b4460 con 0x7fef8c14b190 2024-04-18T10:06:20.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef8b7fe640 1 -- 172.21.15.62:0/339021956 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 330803342 0 0) 0x7fef8c14b620 con 0x7fef8c09b9d0 2024-04-18T10:06:20.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef83fff640 1 --2- 172.21.15.62:0/339021956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef8c14b190 0x7fef8c147ba0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fef740097c0 tx=0x7fef7401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ffbdb4ee02283e server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:20.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef8b7fe640 1 -- 172.21.15.62:0/339021956 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef740b0070 con 0x7fef8c14b190 2024-04-18T10:06:20.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef8b7fe640 1 -- 172.21.15.62:0/339021956 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef740ab600 con 0x7fef8c14b190 2024-04-18T10:06:20.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef8b7fe640 1 -- 172.21.15.62:0/339021956 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef740a4e60 con 0x7fef8c14b190 2024-04-18T10:06:20.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef9239c640 1 -- 172.21.15.62:0/339021956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef8c14b190 msgr2=0x7fef8c147ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef9239c640 1 --2- 172.21.15.62:0/339021956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef8c14b190 0x7fef8c147ba0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fef740097c0 tx=0x7fef7401c3d0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef9239c640 1 -- 172.21.15.62:0/339021956 shutdown_connections 2024-04-18T10:06:20.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef9239c640 1 --2- 172.21.15.62:0/339021956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef8c14b190 0x7fef8c147ba0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef9239c640 1 -- 172.21.15.62:0/339021956 >> 172.21.15.62:0/339021956 conn(0x7fef8c067e10 msgr2=0x7fef8c0bd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:20.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef9239c640 1 -- 172.21.15.62:0/339021956 shutdown_connections 2024-04-18T10:06:20.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.670+0000 7fef9239c640 1 -- 172.21.15.62:0/339021956 wait complete. 2024-04-18T10:06:20.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef9239c640 1 Processor -- start 2024-04-18T10:06:20.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef9239c640 1 -- start start 2024-04-18T10:06:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef9239c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef8c0b3eb0 0x7fef8c0b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef9239c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fef8c0b5e00 0x7fef8c0b6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef9239c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef8c0b4bd0 0x7fef8c0b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef9239c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c156c90 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef9239c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c14c100 con 0x7fef8c0b4bd0 2024-04-18T10:06:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef9239c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fef8c156e10 con 0x7fef8c0b5e00 2024-04-18T10:06:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef8bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef8c0b3eb0 0x7fef8c0b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef8bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef8c0b3eb0 0x7fef8c0b7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:42106/0 (socket says 172.21.15.62:42106) 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef8bfff640 1 -- 172.21.15.62:0/1585953530 learned_addr learned my addr 172.21.15.62:0/1585953530 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef83fff640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fef8c0b5e00 0x7fef8c0b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef90939640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef8c0b4bd0 0x7fef8c0b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef8bfff640 1 -- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fef8c0b5e00 msgr2=0x7fef8c0b6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef8bfff640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fef8c0b5e00 0x7fef8c0b6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef8bfff640 1 -- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef8c0b4bd0 msgr2=0x7fef8c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef8bfff640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef8c0b4bd0 0x7fef8c0b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef8bfff640 1 -- 172.21.15.62:0/1585953530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8c09ebf0 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef90939640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef8c0b4bd0 0x7fef8c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef8bfff640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef8c0b3eb0 0x7fef8c0b7300 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00c480 tx=0x7fef7c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef897fa640 1 -- 172.21.15.62:0/1585953530 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c012830 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef897fa640 1 -- 172.21.15.62:0/1585953530 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef7c017070 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef897fa640 1 -- 172.21.15.62:0/1585953530 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c012c30 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef9239c640 1 -- 172.21.15.62:0/1585953530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef8c0b68a0 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef9239c640 1 -- 172.21.15.62:0/1585953530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef8c09fc20 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.674+0000 7fef9239c640 1 -- 172.21.15.62:0/1585953530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef8c068c00 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.678+0000 7fef897fa640 1 -- 172.21.15.62:0/1585953530 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fef7c01c460 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.678+0000 7fef897fa640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fef6c068380 0x7fef6c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.678+0000 7fef897fa640 1 -- 172.21.15.62:0/1585953530 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fef7c086700 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.678+0000 7fef83fff640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fef6c068380 0x7fef6c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.678+0000 7fef83fff640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fef6c068380 0x7fef6c06a840 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fef740ad000 tx=0x7fef740a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:20.680 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.682+0000 7fef897fa640 1 -- 172.21.15.62:0/1585953530 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fef7c056d50 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.842+0000 7fef9239c640 1 -- 172.21.15.62:0/1585953530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fef8c09b2d0 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.842+0000 7fef897fa640 1 -- 172.21.15.62:0/1585953530 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7fef7c056650 con 0x7fef8c0b3eb0 2024-04-18T10:06:20.843 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-04-18T10:06:20.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.846+0000 7fef9239c640 1 -- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fef6c068380 msgr2=0x7fef6c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.845 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.846+0000 7fef9239c640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fef6c068380 0x7fef6c06a840 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fef740ad000 tx=0x7fef740a6000 comp rx=0 tx=0).stop 2024-04-18T10:06:20.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.846+0000 7fef9239c640 1 -- 172.21.15.62:0/1585953530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef8c0b3eb0 msgr2=0x7fef8c0b7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.846+0000 7fef9239c640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef8c0b3eb0 0x7fef8c0b7300 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00c480 tx=0x7fef7c00c940 comp rx=0 tx=0).stop 2024-04-18T10:06:20.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.846+0000 7fef9239c640 1 -- 172.21.15.62:0/1585953530 shutdown_connections 2024-04-18T10:06:20.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.846+0000 7fef9239c640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fef6c068380 0x7fef6c06a840 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.846+0000 7fef9239c640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fef8c0b4bd0 0x7fef8c0b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.846+0000 7fef9239c640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fef8c0b5e00 0x7fef8c0b6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.846+0000 7fef9239c640 1 --2- 172.21.15.62:0/1585953530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef8c0b3eb0 0x7fef8c0b7300 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.846+0000 7fef9239c640 1 -- 172.21.15.62:0/1585953530 >> 172.21.15.62:0/1585953530 conn(0x7fef8c067e10 msgr2=0x7fef8c0bb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:20.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.846+0000 7fef9239c640 1 -- 172.21.15.62:0/1585953530 shutdown_connections 2024-04-18T10:06:20.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.846+0000 7fef9239c640 1 -- 172.21.15.62:0/1585953530 wait complete. 2024-04-18T10:06:20.862 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args' 2024-04-18T10:06:20.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.978+0000 7fc0e113a640 1 Processor -- start 2024-04-18T10:06:20.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.978+0000 7fc0e113a640 1 -- start start 2024-04-18T10:06:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc0b9d90 0x7fc0dc06d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc0dc09be30 0x7fc0dc06d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc0dc06dc90 0x7fc0dc144f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fc0dc09cd30 con 0x7fc0dc09be30 2024-04-18T10:06:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fc0dc0ba170 con 0x7fc0dc06dc90 2024-04-18T10:06:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc0dc14a060 con 0x7fc0dc0b9d90 2024-04-18T10:06:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0da59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc0dc09be30 0x7fc0dc06d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0da59c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc0dc09be30 0x7fc0dc06d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37542/0 (socket says 172.21.15.62:37542) 2024-04-18T10:06:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0da59c640 1 -- 172.21.15.62:0/2621820333 learned_addr learned my addr 172.21.15.62:0/2621820333 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0da59c640 1 -- 172.21.15.62:0/2621820333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc0b9d90 msgr2=0x7fc0dc06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:20.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0da59c640 1 --2- 172.21.15.62:0/2621820333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc0b9d90 0x7fc0dc06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0da59c640 1 -- 172.21.15.62:0/2621820333 >> v2:172.21.15.145:3300/0 conn(0x7fc0dc06dc90 msgr2=0x7fc0dc144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:20.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0da59c640 1 --2- 172.21.15.62:0/2621820333 >> v2:172.21.15.145:3300/0 conn(0x7fc0dc06dc90 0x7fc0dc144f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0da59c640 1 -- 172.21.15.62:0/2621820333 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0dc149860 con 0x7fc0dc09be30 2024-04-18T10:06:20.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0da59c640 1 --2- 172.21.15.62:0/2621820333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0dc09be30 0x7fc0dc06d750 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc09c700 tx=0x7fc0cc01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43c724167f40609 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0d9d9b640 1 -- 172.21.15.62:0/2621820333 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc0b0070 con 0x7fc0dc09be30 2024-04-18T10:06:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0d9d9b640 1 -- 172.21.15.62:0/2621820333 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0cc0ab600 con 0x7fc0dc09be30 2024-04-18T10:06:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0d9d9b640 1 -- 172.21.15.62:0/2621820333 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc0ab9d0 con 0x7fc0dc09be30 2024-04-18T10:06:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 -- 172.21.15.62:0/2621820333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0dc09be30 msgr2=0x7fc0dc06d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 --2- 172.21.15.62:0/2621820333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0dc09be30 0x7fc0dc06d750 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc0dc09c700 tx=0x7fc0cc01c400 comp rx=0 tx=0).stop 2024-04-18T10:06:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 -- 172.21.15.62:0/2621820333 shutdown_connections 2024-04-18T10:06:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 --2- 172.21.15.62:0/2621820333 >> v2:172.21.15.145:3300/0 conn(0x7fc0dc06dc90 0x7fc0dc144f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 --2- 172.21.15.62:0/2621820333 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0dc09be30 0x7fc0dc06d750 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 --2- 172.21.15.62:0/2621820333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc0b9d90 0x7fc0dc06d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 -- 172.21.15.62:0/2621820333 >> 172.21.15.62:0/2621820333 conn(0x7fc0dc06c0d0 msgr2=0x7fc0dc0b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:20.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 -- 172.21.15.62:0/2621820333 shutdown_connections 2024-04-18T10:06:20.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.982+0000 7fc0e113a640 1 -- 172.21.15.62:0/2621820333 wait complete. 2024-04-18T10:06:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0e113a640 1 Processor -- start 2024-04-18T10:06:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0e113a640 1 -- start start 2024-04-18T10:06:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0e113a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc06dc90 0x7fc0dc1e3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0e113a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0dc09be30 0x7fc0dc147ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0e113a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0dc0b9d90 0x7fc0dc1481e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0e113a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc145de0 con 0x7fc0dc09be30 2024-04-18T10:06:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0e113a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fc0dc145c60 con 0x7fc0dc0b9d90 2024-04-18T10:06:20.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0e113a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc0dc145f60 con 0x7fc0dc06dc90 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0da59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0dc09be30 0x7fc0dc147ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0da59c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0dc09be30 0x7fc0dc147ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37546/0 (socket says 172.21.15.62:37546) 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0da59c640 1 -- 172.21.15.62:0/4027562196 learned_addr learned my addr 172.21.15.62:0/4027562196 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0dad9d640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc06dc90 0x7fc0dc1e3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0db59e640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0dc0b9d90 0x7fc0dc1481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0da59c640 1 -- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc06dc90 msgr2=0x7fc0dc1e3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0da59c640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc06dc90 0x7fc0dc1e3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0da59c640 1 -- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0dc0b9d90 msgr2=0x7fc0dc1481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0da59c640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0dc0b9d90 0x7fc0dc1481e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0da59c640 1 -- 172.21.15.62:0/4027562196 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0dc0b0c10 con 0x7fc0dc09be30 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0db59e640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0dc0b9d90 0x7fc0dc1481e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0dad9d640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc06dc90 0x7fc0dc1e3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0da59c640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0dc09be30 0x7fc0dc147ca0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0ae040 tx=0x7fc0cc01ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0c37fe640 1 -- 172.21.15.62:0/4027562196 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc0ba020 con 0x7fc0dc09be30 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0c37fe640 1 -- 172.21.15.62:0/4027562196 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0cc0b6640 con 0x7fc0dc09be30 2024-04-18T10:06:20.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0e113a640 1 -- 172.21.15.62:0/4027562196 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0dc1e4580 con 0x7fc0dc09be30 2024-04-18T10:06:20.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0c37fe640 1 -- 172.21.15.62:0/4027562196 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc0b0040 con 0x7fc0dc09be30 2024-04-18T10:06:20.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0e113a640 1 -- 172.21.15.62:0/4027562196 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0dc0736a0 con 0x7fc0dc09be30 2024-04-18T10:06:20.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.986+0000 7fc0e113a640 1 -- 172.21.15.62:0/4027562196 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0dc068b90 con 0x7fc0dc09be30 2024-04-18T10:06:20.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.990+0000 7fc0c37fe640 1 -- 172.21.15.62:0/4027562196 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc0cc0bd020 con 0x7fc0dc09be30 2024-04-18T10:06:20.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.990+0000 7fc0c37fe640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc0c8068380 0x7fc0c806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:20.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.990+0000 7fc0c37fe640 1 -- 172.21.15.62:0/4027562196 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc0cc120cd0 con 0x7fc0dc09be30 2024-04-18T10:06:20.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.990+0000 7fc0dad9d640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc0c8068380 0x7fc0c806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:20.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.990+0000 7fc0dad9d640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc0c8068380 0x7fc0c806a840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000a870 tx=0x7fc0d0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:20.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:20.994+0000 7fc0c37fe640 1 -- 172.21.15.62:0/4027562196 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc0cc0c4030 con 0x7fc0dc09be30 2024-04-18T10:06:21.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.206+0000 7fc0e113a640 1 -- 172.21.15.62:0/4027562196 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args"]} v 0) -- 0x7fc0dc144aa0 con 0x7fc0dc09be30 2024-04-18T10:06:21.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.330+0000 7fc0c37fe640 1 -- 172.21.15.62:0/4027562196 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args"]}]=0 v45) ==== 137+0+0 (secure 0 0 0) 0x7fc0cc016110 con 0x7fc0dc09be30 2024-04-18T10:06:21.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.334+0000 7fc0e113a640 1 -- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc0c8068380 msgr2=0x7fc0c806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:21.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.334+0000 7fc0e113a640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc0c8068380 0x7fc0c806a840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000a870 tx=0x7fc0d0008040 comp rx=0 tx=0).stop 2024-04-18T10:06:21.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.334+0000 7fc0e113a640 1 -- 172.21.15.62:0/4027562196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0dc09be30 msgr2=0x7fc0dc147ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:21.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.334+0000 7fc0e113a640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0dc09be30 0x7fc0dc147ca0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc0ae040 tx=0x7fc0cc01ac40 comp rx=0 tx=0).stop 2024-04-18T10:06:21.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.334+0000 7fc0e113a640 1 -- 172.21.15.62:0/4027562196 shutdown_connections 2024-04-18T10:06:21.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.334+0000 7fc0e113a640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc0c8068380 0x7fc0c806a840 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.334+0000 7fc0e113a640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0dc0b9d90 0x7fc0dc1481e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.334+0000 7fc0e113a640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0dc09be30 0x7fc0dc147ca0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.334+0000 7fc0e113a640 1 --2- 172.21.15.62:0/4027562196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0dc06dc90 0x7fc0dc1e3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.334+0000 7fc0e113a640 1 -- 172.21.15.62:0/4027562196 >> 172.21.15.62:0/4027562196 conn(0x7fc0dc06c0d0 msgr2=0x7fc0dc0aeab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:21.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.334+0000 7fc0e113a640 1 -- 172.21.15.62:0/4027562196 shutdown_connections 2024-04-18T10:06:21.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.334+0000 7fc0e113a640 1 -- 172.21.15.62:0/4027562196 wait complete. 2024-04-18T10:06:21.351 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-18T10:06:21.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.482+0000 7f004204f640 1 Processor -- start 2024-04-18T10:06:21.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.482+0000 7f004204f640 1 -- start start 2024-04-18T10:06:21.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f004204f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f003c146690 0x7f003c09e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:21.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f004204f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c06e490 0x7f003c09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:21.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f004204f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f003c06d4f0 con 0x7f003c0b4c90 2024-04-18T10:06:21.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f004204f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f003c06d2a0 con 0x7f003c146690 2024-04-18T10:06:21.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f004204f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f003c06d7d0 con 0x7f003c06e490 2024-04-18T10:06:21.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003b7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f003c0b4c90 0x7f003c145f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35272/0 (socket says 172.21.15.62:35272) 2024-04-18T10:06:21.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003b7fe640 1 -- 172.21.15.62:0/3068128007 learned_addr learned my addr 172.21.15.62:0/3068128007 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:21.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003affd640 1 --2- 172.21.15.62:0/3068128007 >> v2:172.21.15.145:3300/0 conn(0x7f003c146690 0x7f003c09e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1423781271 0 0) 0x7f003c06d4f0 con 0x7f003c0b4c90 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0028003530 con 0x7f003c0b4c90 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003bfff640 1 --2- 172.21.15.62:0/3068128007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c06e490 0x7f003c09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 860373804 0 0) 0x7f0028003530 con 0x7f003c0b4c90 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c06e490 msgr2=0x7f003c09e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 --2- 172.21.15.62:0/3068128007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c06e490 0x7f003c09e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 >> v2:172.21.15.145:3300/0 conn(0x7f003c146690 msgr2=0x7f003c09e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 --2- 172.21.15.62:0/3068128007 >> v2:172.21.15.145:3300/0 conn(0x7f003c146690 0x7f003c09e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003c09c570 con 0x7f003c0b4c90 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f002c001680 con 0x7f003c0b4c90 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 >> v1:172.21.15.62:6789/0 conn(0x7f003c0b4c90 legacy=0x7f003c145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 --2- 172.21.15.62:0/3068128007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c06e490 0x7f00280039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:21.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 --2- 172.21.15.62:0/3068128007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0028003f40 0x7f0028004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:21.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 --2- 172.21.15.62:0/3068128007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028004d50 0x7f00280051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:21.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f003c06d2a0 con 0x7f0028003f40 2024-04-18T10:06:21.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f003c06d7d0 con 0x7f0028004d50 2024-04-18T10:06:21.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f002800ba20 con 0x7f003c06e490 2024-04-18T10:06:21.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003affd640 1 --2- 172.21.15.62:0/3068128007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0028003f40 0x7f0028004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:21.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003bfff640 1 --2- 172.21.15.62:0/3068128007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028004d50 0x7f00280051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:21.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003b7fe640 1 --2- 172.21.15.62:0/3068128007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c06e490 0x7f00280039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:21.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003affd640 1 -- 172.21.15.62:0/3068128007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c06e490 msgr2=0x7f00280039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:21.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003affd640 1 --2- 172.21.15.62:0/3068128007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c06e490 0x7f00280039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003affd640 1 -- 172.21.15.62:0/3068128007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028004d50 msgr2=0x7f00280051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:21.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003affd640 1 --2- 172.21.15.62:0/3068128007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028004d50 0x7f00280051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.487 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.486+0000 7f003affd640 1 -- 172.21.15.62:0/3068128007 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0028003530 con 0x7f0028003f40 2024-04-18T10:06:21.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003affd640 1 --2- 172.21.15.62:0/3068128007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0028003f40 0x7f0028004810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0024009b10 tx=0x7f002401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7617d3e0d451236b server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:21.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00240b2070 con 0x7f0028003f40 2024-04-18T10:06:21.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00240a5510 con 0x7f0028003f40 2024-04-18T10:06:21.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003a7fc640 1 -- 172.21.15.62:0/3068128007 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00240096b0 con 0x7f0028003f40 2024-04-18T10:06:21.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 -- 172.21.15.62:0/3068128007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0028003f40 msgr2=0x7f0028004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:21.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 --2- 172.21.15.62:0/3068128007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0028003f40 0x7f0028004810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0024009b10 tx=0x7f002401cab0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003bfff640 1 -- 172.21.15.62:0/3068128007 reap_dead start 2024-04-18T10:06:21.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 -- 172.21.15.62:0/3068128007 shutdown_connections 2024-04-18T10:06:21.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 -- 172.21.15.62:0/3068128007 >> 172.21.15.62:0/3068128007 conn(0x7f003c0b9690 msgr2=0x7f003c0b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:21.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 -- 172.21.15.62:0/3068128007 shutdown_connections 2024-04-18T10:06:21.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 -- 172.21.15.62:0/3068128007 wait complete. 2024-04-18T10:06:21.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 Processor -- start 2024-04-18T10:06:21.490 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 -- start start 2024-04-18T10:06:21.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028003f40 0x7f003c072e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:21.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c0b2650 0x7f003c0b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:21.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f003c0b0f50 0x7f003c0ae4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:21.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f003c150da0 con 0x7f003c0b0f50 2024-04-18T10:06:21.491 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f003c150c20 con 0x7f0028003f40 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f003c150f20 con 0x7f003c0b2650 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f003c0b0f50 0x7f003c0ae4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f003c0b0f50 0x7f003c0ae4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37566/0 (socket says 172.21.15.62:37566) 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003bfff640 1 -- 172.21.15.62:0/4242574935 learned_addr learned my addr 172.21.15.62:0/4242574935 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003affd640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c0b2650 0x7f003c0b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003b7fe640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028003f40 0x7f003c072e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003bfff640 1 -- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c0b2650 msgr2=0x7f003c0b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003bfff640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c0b2650 0x7f003c0b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003bfff640 1 -- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028003f40 msgr2=0x7f003c072e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003bfff640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028003f40 0x7f003c072e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003bfff640 1 -- 172.21.15.62:0/4242574935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f003c1438b0 con 0x7f003c0b0f50 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003b7fe640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028003f40 0x7f003c072e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f003bfff640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f003c0b0f50 0x7f003c0ae4e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f003c06e920 tx=0x7f003400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f0033fff640 1 -- 172.21.15.62:0/4242574935 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003401c070 con 0x7f003c0b0f50 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f0033fff640 1 -- 172.21.15.62:0/4242574935 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0034017400 con 0x7f003c0b0f50 2024-04-18T10:06:21.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 -- 172.21.15.62:0/4242574935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f003c0b13b0 con 0x7f003c0b0f50 2024-04-18T10:06:21.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f0033fff640 1 -- 172.21.15.62:0/4242574935 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00340177f0 con 0x7f003c0b0f50 2024-04-18T10:06:21.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.490+0000 7f004204f640 1 -- 172.21.15.62:0/4242574935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f003c0b2da0 con 0x7f003c0b0f50 2024-04-18T10:06:21.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.494+0000 7f004204f640 1 -- 172.21.15.62:0/4242574935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f003c068c00 con 0x7f003c0b0f50 2024-04-18T10:06:21.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.494+0000 7f0033fff640 1 -- 172.21.15.62:0/4242574935 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f003401f070 con 0x7f003c0b0f50 2024-04-18T10:06:21.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.494+0000 7f0033fff640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f001c0683b0 0x7f001c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:21.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.494+0000 7f0033fff640 1 -- 172.21.15.62:0/4242574935 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0034086aa0 con 0x7f003c0b0f50 2024-04-18T10:06:21.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.494+0000 7f003b7fe640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f001c0683b0 0x7f001c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:21.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.494+0000 7f003b7fe640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f001c0683b0 0x7f001c06a870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f002c000c10 tx=0x7f002c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:21.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.498+0000 7f0033fff640 1 -- 172.21.15.62:0/4242574935 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0034012070 con 0x7f003c0b0f50 2024-04-18T10:06:21.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.658+0000 7f004204f640 1 -- 172.21.15.62:0/4242574935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f003c0b44c0 con 0x7f003c0b0f50 2024-04-18T10:06:21.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.662+0000 7f0033fff640 1 -- 172.21.15.62:0/4242574935 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f00340570f0 con 0x7f003c0b0f50 2024-04-18T10:06:21.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.666+0000 7f004204f640 1 -- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f001c0683b0 msgr2=0x7f001c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:21.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.666+0000 7f004204f640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f001c0683b0 0x7f001c06a870 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f002c000c10 tx=0x7f002c009040 comp rx=0 tx=0).stop 2024-04-18T10:06:21.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.666+0000 7f004204f640 1 -- 172.21.15.62:0/4242574935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f003c0b0f50 msgr2=0x7f003c0ae4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.666+0000 7f004204f640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f003c0b0f50 0x7f003c0ae4e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f003c06e920 tx=0x7f003400f4a0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.666+0000 7f004204f640 1 -- 172.21.15.62:0/4242574935 shutdown_connections 2024-04-18T10:06:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.666+0000 7f004204f640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f001c0683b0 0x7f001c06a870 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.666+0000 7f004204f640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f003c0b0f50 0x7f003c0ae4e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.666+0000 7f004204f640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f003c0b2650 0x7f003c0b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.666+0000 7f004204f640 1 --2- 172.21.15.62:0/4242574935 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0028003f40 0x7f003c072e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.666+0000 7f004204f640 1 -- 172.21.15.62:0/4242574935 >> 172.21.15.62:0/4242574935 conn(0x7f003c0b9690 msgr2=0x7f003c09cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.666+0000 7f004204f640 1 -- 172.21.15.62:0/4242574935 shutdown_connections 2024-04-18T10:06:21.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:21.666+0000 7f004204f640 1 -- 172.21.15.62:0/4242574935 wait complete. 2024-04-18T10:06:21.679 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-18T10:06:21.679 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-18T10:06:21.732 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-18T10:06:21.789 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-18T10:06:21.789 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:06:21.789 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-18T10:06:21.794 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:06:21.794 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-18T10:06:21.840 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:22.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f84372b1640 1 Processor -- start 2024-04-18T10:06:22.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f84372b1640 1 -- start start 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f84372b1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84300b9d90 0x7f843006d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f84372b1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f843009be30 0x7f843006d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f84372b1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84300ba170 con 0x7f843006dc90 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f84372b1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f843009cd30 con 0x7f843009be30 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f84372b1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f843014a060 con 0x7f84300b9d90 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f843484c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f843009be30 0x7f843006d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f843584e640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f843006dc90 0x7f8430144f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35274/0 (socket says 172.21.15.62:35274) 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f843584e640 1 -- 172.21.15.62:0/1652639049 learned_addr learned my addr 172.21.15.62:0/1652639049 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f843504d640 1 --2- 172.21.15.62:0/1652639049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84300b9d90 0x7f843006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f843484c640 1 -- 172.21.15.62:0/1652639049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84300b9d90 msgr2=0x7f843006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f843484c640 1 --2- 172.21.15.62:0/1652639049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84300b9d90 0x7f843006d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f843484c640 1 -- 172.21.15.62:0/1652639049 >> v1:172.21.15.62:6789/0 conn(0x7f843006dc90 legacy=0x7f8430144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f842ffff640 1 -- 172.21.15.62:0/1652639049 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4282805971 0 0) 0x7f84300ba170 con 0x7f843006dc90 2024-04-18T10:06:22.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.010+0000 7f843484c640 1 -- 172.21.15.62:0/1652639049 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8430149860 con 0x7f843009be30 2024-04-18T10:06:22.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.014+0000 7f843484c640 1 --2- 172.21.15.62:0/1652639049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f843009be30 0x7f843006d750 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8424009d90 tx=0x7f842401c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a93ee4541ee8a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:22.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.014+0000 7f842ffff640 1 -- 172.21.15.62:0/1652639049 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84240b2070 con 0x7f843009be30 2024-04-18T10:06:22.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.014+0000 7f842ffff640 1 -- 172.21.15.62:0/1652639049 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84240a5820 con 0x7f843009be30 2024-04-18T10:06:22.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.014+0000 7f842ffff640 1 -- 172.21.15.62:0/1652639049 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84240a5bf0 con 0x7f843009be30 2024-04-18T10:06:22.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.014+0000 7f84372b1640 1 -- 172.21.15.62:0/1652639049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f843009be30 msgr2=0x7f843006d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.014+0000 7f84372b1640 1 --2- 172.21.15.62:0/1652639049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f843009be30 0x7f843006d750 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8424009d90 tx=0x7f842401c850 comp rx=0 tx=0).stop 2024-04-18T10:06:22.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.014+0000 7f84372b1640 1 -- 172.21.15.62:0/1652639049 shutdown_connections 2024-04-18T10:06:22.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.014+0000 7f84372b1640 1 --2- 172.21.15.62:0/1652639049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f843009be30 0x7f843006d750 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.014+0000 7f84372b1640 1 --2- 172.21.15.62:0/1652639049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84300b9d90 0x7f843006d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.014+0000 7f84372b1640 1 -- 172.21.15.62:0/1652639049 >> 172.21.15.62:0/1652639049 conn(0x7f843006c0d0 msgr2=0x7f84300b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:22.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.014+0000 7f84372b1640 1 -- 172.21.15.62:0/1652639049 shutdown_connections 2024-04-18T10:06:22.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.014+0000 7f84372b1640 1 -- 172.21.15.62:0/1652639049 wait complete. 2024-04-18T10:06:22.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f84372b1640 1 Processor -- start 2024-04-18T10:06:22.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f84372b1640 1 -- start start 2024-04-18T10:06:22.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f84372b1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f843006dc90 0x7f8430144740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f84372b1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f843009be30 0x7f84301e3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f84372b1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84300b9d90 0x7f8430140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f84372b1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8430145fb0 con 0x7f843006dc90 2024-04-18T10:06:22.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f84372b1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f8430145e30 con 0x7f84300b9d90 2024-04-18T10:06:22.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f84372b1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8430146130 con 0x7f843009be30 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f843504d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f843006dc90 0x7f8430144740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f843504d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f843006dc90 0x7f8430144740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37588/0 (socket says 172.21.15.62:37588) 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f843504d640 1 -- 172.21.15.62:0/1198524314 learned_addr learned my addr 172.21.15.62:0/1198524314 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f843504d640 1 -- 172.21.15.62:0/1198524314 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f843009be30 msgr2=0x7f84301e3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f843504d640 1 --2- 172.21.15.62:0/1198524314 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f843009be30 0x7f84301e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f843504d640 1 -- 172.21.15.62:0/1198524314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84300b9d90 msgr2=0x7f8430140c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f843504d640 1 --2- 172.21.15.62:0/1198524314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84300b9d90 0x7f8430140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f843504d640 1 -- 172.21.15.62:0/1198524314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84301436a0 con 0x7f843006dc90 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f843504d640 1 --2- 172.21.15.62:0/1198524314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f843006dc90 0x7f8430144740 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f842000d620 tx=0x7f842000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f842dffb640 1 -- 172.21.15.62:0/1198524314 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842000b620 con 0x7f843006dc90 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f842dffb640 1 -- 172.21.15.62:0/1198524314 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8420019070 con 0x7f843006dc90 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f842dffb640 1 -- 172.21.15.62:0/1198524314 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f842000b9e0 con 0x7f843006dc90 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f84372b1640 1 -- 172.21.15.62:0/1198524314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8430143980 con 0x7f843006dc90 2024-04-18T10:06:22.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f84372b1640 1 -- 172.21.15.62:0/1198524314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8430142a90 con 0x7f843006dc90 2024-04-18T10:06:22.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f84372b1640 1 -- 172.21.15.62:0/1198524314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8430144390 con 0x7f843006dc90 2024-04-18T10:06:22.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f842dffb640 1 -- 172.21.15.62:0/1198524314 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f842000bb80 con 0x7f843006dc90 2024-04-18T10:06:22.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.018+0000 7f842dffb640 1 --2- 172.21.15.62:0/1198524314 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f841c068380 0x7f841c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.022+0000 7f842dffb640 1 -- 172.21.15.62:0/1198524314 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8420086d60 con 0x7f843006dc90 2024-04-18T10:06:22.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.022+0000 7f843484c640 1 --2- 172.21.15.62:0/1198524314 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f841c068380 0x7f841c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.022+0000 7f843484c640 1 --2- 172.21.15.62:0/1198524314 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f841c068380 0x7f841c06a840 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f84240af000 tx=0x7f8424004ac0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:22.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.026+0000 7f842dffb640 1 -- 172.21.15.62:0/1198524314 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8430144390 con 0x7f843006dc90 2024-04-18T10:06:22.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.182+0000 7f84372b1640 1 -- 172.21.15.62:0/1198524314 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8430068b90 con 0x7f843006dc90 2024-04-18T10:06:22.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.182+0000 7f842dffb640 1 -- 172.21.15.62:0/1198524314 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f84200573b0 con 0x7f843006dc90 2024-04-18T10:06:22.183 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:22.183 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:22.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.186+0000 7f84372b1640 1 -- 172.21.15.62:0/1198524314 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f841c068380 msgr2=0x7f841c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.186+0000 7f84372b1640 1 --2- 172.21.15.62:0/1198524314 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f841c068380 0x7f841c06a840 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f84240af000 tx=0x7f8424004ac0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.186+0000 7f84372b1640 1 -- 172.21.15.62:0/1198524314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f843006dc90 msgr2=0x7f8430144740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.186+0000 7f84372b1640 1 --2- 172.21.15.62:0/1198524314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f843006dc90 0x7f8430144740 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f842000d620 tx=0x7f842000dae0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.186+0000 7f84372b1640 1 -- 172.21.15.62:0/1198524314 shutdown_connections 2024-04-18T10:06:22.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.186+0000 7f84372b1640 1 --2- 172.21.15.62:0/1198524314 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f841c068380 0x7f841c06a840 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.186+0000 7f84372b1640 1 --2- 172.21.15.62:0/1198524314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84300b9d90 0x7f8430140c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.186+0000 7f84372b1640 1 --2- 172.21.15.62:0/1198524314 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f843009be30 0x7f84301e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.186+0000 7f84372b1640 1 --2- 172.21.15.62:0/1198524314 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f843006dc90 0x7f8430144740 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.186+0000 7f84372b1640 1 -- 172.21.15.62:0/1198524314 >> 172.21.15.62:0/1198524314 conn(0x7f843006c0d0 msgr2=0x7f843006d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:22.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.186+0000 7f84372b1640 1 -- 172.21.15.62:0/1198524314 shutdown_connections 2024-04-18T10:06:22.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.186+0000 7f84372b1640 1 -- 172.21.15.62:0/1198524314 wait complete. 2024-04-18T10:06:22.200 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-18T10:05:56.782649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-18T10:05:57.541623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:22.201 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:22.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.322+0000 7f4697a33640 1 Processor -- start 2024-04-18T10:06:22.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.322+0000 7f4697a33640 1 -- start start 2024-04-18T10:06:22.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f4697a33640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46900b4c90 0x7f4690145f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f4697a33640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f469006e490 0x7f46901464c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f4697a33640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f469006d4f0 con 0x7f469009e090 2024-04-18T10:06:22.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f4697a33640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f469006d2a0 con 0x7f469006e490 2024-04-18T10:06:22.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f4697a33640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f469006d7d0 con 0x7f46900b4c90 2024-04-18T10:06:22.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f4694fce640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f469006e490 0x7f46901464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f46957cf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46900b4c90 0x7f4690145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f46957cf640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46900b4c90 0x7f4690145f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:57822/0 (socket says 172.21.15.62:57822) 2024-04-18T10:06:22.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f46957cf640 1 -- 172.21.15.62:0/3486947701 learned_addr learned my addr 172.21.15.62:0/3486947701 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:22.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f46957cf640 1 -- 172.21.15.62:0/3486947701 >> v2:172.21.15.145:3300/0 conn(0x7f469006e490 msgr2=0x7f46901464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:22.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f46957cf640 1 --2- 172.21.15.62:0/3486947701 >> v2:172.21.15.145:3300/0 conn(0x7f469006e490 0x7f46901464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f46957cf640 1 -- 172.21.15.62:0/3486947701 >> v1:172.21.15.145:6789/0 conn(0x7f469009e090 legacy=0x7f469009e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f46957cf640 1 -- 172.21.15.62:0/3486947701 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469009c570 con 0x7f46900b4c90 2024-04-18T10:06:22.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f4694fce640 1 --2- 172.21.15.62:0/3486947701 >> v2:172.21.15.145:3300/0 conn(0x7f469006e490 0x7f46901464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:06:22.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f46957cf640 1 --2- 172.21.15.62:0/3486947701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46900b4c90 0x7f4690145f80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4684009870 tx=0x7f468401c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6092a4a4f4d20f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:22.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f4683fff640 1 -- 172.21.15.62:0/3486947701 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46840b1070 con 0x7f46900b4c90 2024-04-18T10:06:22.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f4683fff640 1 -- 172.21.15.62:0/3486947701 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46840a8050 con 0x7f46900b4c90 2024-04-18T10:06:22.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.326+0000 7f4683fff640 1 -- 172.21.15.62:0/3486947701 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46840ac860 con 0x7f46900b4c90 2024-04-18T10:06:22.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 -- 172.21.15.62:0/3486947701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46900b4c90 msgr2=0x7f4690145f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 --2- 172.21.15.62:0/3486947701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46900b4c90 0x7f4690145f80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4684009870 tx=0x7f468401c4d0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 -- 172.21.15.62:0/3486947701 shutdown_connections 2024-04-18T10:06:22.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 --2- 172.21.15.62:0/3486947701 >> v2:172.21.15.145:3300/0 conn(0x7f469006e490 0x7f46901464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 --2- 172.21.15.62:0/3486947701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46900b4c90 0x7f4690145f80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 -- 172.21.15.62:0/3486947701 >> 172.21.15.62:0/3486947701 conn(0x7f46900b9690 msgr2=0x7f46900b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:22.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 -- 172.21.15.62:0/3486947701 shutdown_connections 2024-04-18T10:06:22.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 -- 172.21.15.62:0/3486947701 wait complete. 2024-04-18T10:06:22.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 Processor -- start 2024-04-18T10:06:22.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 -- start start 2024-04-18T10:06:22.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f469006e490 0x7f46900726a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f469009e090 0x7f46901420f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46900b4c90 0x7f469014f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4690152fc0 con 0x7f469006e490 2024-04-18T10:06:22.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f4690152e40 con 0x7f469009e090 2024-04-18T10:06:22.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4697a33640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f4690153140 con 0x7f46900b4c90 2024-04-18T10:06:22.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f46957cf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f469006e490 0x7f46900726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f46957cf640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f469006e490 0x7f46900726a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37600/0 (socket says 172.21.15.62:37600) 2024-04-18T10:06:22.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f46957cf640 1 -- 172.21.15.62:0/3773677577 learned_addr learned my addr 172.21.15.62:0/3773677577 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:22.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4695fd0640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46900b4c90 0x7f469014f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f4694fce640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f469009e090 0x7f46901420f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f46957cf640 1 -- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46900b4c90 msgr2=0x7f469014f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f46957cf640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46900b4c90 0x7f469014f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f46957cf640 1 -- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f469009e090 msgr2=0x7f46901420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f46957cf640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f469009e090 0x7f46901420f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f46957cf640 1 -- 172.21.15.62:0/3773677577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46901438d0 con 0x7f469006e490 2024-04-18T10:06:22.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.330+0000 7f46957cf640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f469006e490 0x7f46900726a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f468401a100 tx=0x7f4684019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:22.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.334+0000 7f4681ffb640 1 -- 172.21.15.62:0/3773677577 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46840bb020 con 0x7f469006e490 2024-04-18T10:06:22.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.334+0000 7f4681ffb640 1 -- 172.21.15.62:0/3773677577 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46840ace10 con 0x7f469006e490 2024-04-18T10:06:22.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.334+0000 7f4697a33640 1 -- 172.21.15.62:0/3773677577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4690147ea0 con 0x7f469006e490 2024-04-18T10:06:22.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.334+0000 7f4681ffb640 1 -- 172.21.15.62:0/3773677577 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46840b1040 con 0x7f469006e490 2024-04-18T10:06:22.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.334+0000 7f4697a33640 1 -- 172.21.15.62:0/3773677577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4690073290 con 0x7f469006e490 2024-04-18T10:06:22.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.334+0000 7f4697a33640 1 -- 172.21.15.62:0/3773677577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4690068c00 con 0x7f469006e490 2024-04-18T10:06:22.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.334+0000 7f4681ffb640 1 -- 172.21.15.62:0/3773677577 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f46840be020 con 0x7f469006e490 2024-04-18T10:06:22.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.338+0000 7f4681ffb640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4674068380 0x7f467406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.338+0000 7f4681ffb640 1 -- 172.21.15.62:0/3773677577 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4684120ab0 con 0x7f469006e490 2024-04-18T10:06:22.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.338+0000 7f4694fce640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4674068380 0x7f467406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.338+0000 7f4694fce640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4674068380 0x7f467406a840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f468c009bd0 tx=0x7f468c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:22.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.342+0000 7f4681ffb640 1 -- 172.21.15.62:0/3773677577 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f46840c5030 con 0x7f469006e490 2024-04-18T10:06:22.504 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.506+0000 7f4697a33640 1 -- 172.21.15.62:0/3773677577 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f46900b44c0 con 0x7f469006e490 2024-04-18T10:06:22.505 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.506+0000 7f4681ffb640 1 -- 172.21.15.62:0/3773677577 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f4684016150 con 0x7f469006e490 2024-04-18T10:06:22.505 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:22.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.510+0000 7f4697a33640 1 -- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4674068380 msgr2=0x7f467406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.510+0000 7f4697a33640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4674068380 0x7f467406a840 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f468c009bd0 tx=0x7f468c00b040 comp rx=0 tx=0).stop 2024-04-18T10:06:22.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.510+0000 7f4697a33640 1 -- 172.21.15.62:0/3773677577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f469006e490 msgr2=0x7f46900726a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.510+0000 7f4697a33640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f469006e490 0x7f46900726a0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f468401a100 tx=0x7f4684019270 comp rx=0 tx=0).stop 2024-04-18T10:06:22.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.510+0000 7f4697a33640 1 -- 172.21.15.62:0/3773677577 shutdown_connections 2024-04-18T10:06:22.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.510+0000 7f4697a33640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4674068380 0x7f467406a840 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.508 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.510+0000 7f4697a33640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46900b4c90 0x7f469014f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.510+0000 7f4697a33640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f469009e090 0x7f46901420f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.510+0000 7f4697a33640 1 --2- 172.21.15.62:0/3773677577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f469006e490 0x7f46900726a0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.510+0000 7f4697a33640 1 -- 172.21.15.62:0/3773677577 >> 172.21.15.62:0/3773677577 conn(0x7f46900b9690 msgr2=0x7f4690144350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:22.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.510+0000 7f4697a33640 1 -- 172.21.15.62:0/3773677577 shutdown_connections 2024-04-18T10:06:22.509 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.510+0000 7f4697a33640 1 -- 172.21.15.62:0/3773677577 wait complete. 2024-04-18T10:06:22.525 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-18T10:06:22.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.646+0000 7fbdefaa5640 1 Processor -- start 2024-04-18T10:06:22.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.646+0000 7fbdefaa5640 1 -- start start 2024-04-18T10:06:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.646+0000 7fbdefaa5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbde8067bf0 0x7fbde813fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.646+0000 7fbdefaa5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde806ed90 con 0x7fbde8067810 2024-04-18T10:06:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.646+0000 7fbdefaa5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbde8145c50 con 0x7fbde8146730 2024-04-18T10:06:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.646+0000 7fbdefaa5640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fbde81462a0 con 0x7fbde8067bf0 2024-04-18T10:06:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.646+0000 7fbded841640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbde8067810 0x7fbde806da80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35280/0 (socket says 172.21.15.62:35280) 2024-04-18T10:06:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.646+0000 7fbded841640 1 -- 172.21.15.62:0/2393315537 learned_addr learned my addr 172.21.15.62:0/2393315537 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:22.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.646+0000 7fbdee042640 1 --2- 172.21.15.62:0/2393315537 >> v2:172.21.15.62:3300/0 conn(0x7fbde8067bf0 0x7fbde813fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.646+0000 7fbdee042640 1 -- 172.21.15.62:0/2393315537 >> v1:172.21.15.145:6789/0 conn(0x7fbde8146730 legacy=0x7fbde80b5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:22.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.646+0000 7fbdee042640 1 -- 172.21.15.62:0/2393315537 >> v1:172.21.15.62:6789/0 conn(0x7fbde8067810 legacy=0x7fbde806da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.646+0000 7fbdee042640 1 -- 172.21.15.62:0/2393315537 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde8145f30 con 0x7fbde8067bf0 2024-04-18T10:06:22.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdee042640 1 --2- 172.21.15.62:0/2393315537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbde8067bf0 0x7fbde813fab0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009d90 tx=0x7fbde001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e660a4100f4333 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:22.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdec83f640 1 -- 172.21.15.62:0/2393315537 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde00b2070 con 0x7fbde8067bf0 2024-04-18T10:06:22.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdec83f640 1 -- 172.21.15.62:0/2393315537 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbde00a57f0 con 0x7fbde8067bf0 2024-04-18T10:06:22.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdec83f640 1 -- 172.21.15.62:0/2393315537 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbde00a5bc0 con 0x7fbde8067bf0 2024-04-18T10:06:22.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 -- 172.21.15.62:0/2393315537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbde8067bf0 msgr2=0x7fbde813fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 --2- 172.21.15.62:0/2393315537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbde8067bf0 0x7fbde813fab0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009d90 tx=0x7fbde001c930 comp rx=0 tx=0).stop 2024-04-18T10:06:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 -- 172.21.15.62:0/2393315537 shutdown_connections 2024-04-18T10:06:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 --2- 172.21.15.62:0/2393315537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbde8067bf0 0x7fbde813fab0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 -- 172.21.15.62:0/2393315537 >> 172.21.15.62:0/2393315537 conn(0x7fbde80ba870 msgr2=0x7fbde80b67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 -- 172.21.15.62:0/2393315537 shutdown_connections 2024-04-18T10:06:22.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 -- 172.21.15.62:0/2393315537 wait complete. 2024-04-18T10:06:22.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 Processor -- start 2024-04-18T10:06:22.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 -- start start 2024-04-18T10:06:22.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbde8067810 0x7fbde80b9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbde8067bf0 0x7fbde8133da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbde8146730 0x7fbde8139190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbde8153e00 con 0x7fbde8067810 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fbde8071c30 con 0x7fbde8146730 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdefaa5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fbde8153f80 con 0x7fbde8067bf0 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbded841640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbde8067810 0x7fbde80b9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbded841640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbde8067810 0x7fbde80b9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37626/0 (socket says 172.21.15.62:37626) 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbded841640 1 -- 172.21.15.62:0/1235747769 learned_addr learned my addr 172.21.15.62:0/1235747769 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbded040640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbde8067bf0 0x7fbde8133da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbded841640 1 -- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbde8067bf0 msgr2=0x7fbde8133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbdee042640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbde8146730 0x7fbde8139190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbded841640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbde8067bf0 0x7fbde8133da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbded841640 1 -- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbde8146730 msgr2=0x7fbde8139190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbded841640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbde8146730 0x7fbde8139190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.650+0000 7fbded841640 1 -- 172.21.15.62:0/1235747769 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde81335e0 con 0x7fbde8067810 2024-04-18T10:06:22.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.654+0000 7fbded841640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbde8067810 0x7fbde80b9210 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800c420 tx=0x7fbdd800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:22.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.654+0000 7fbdde7fc640 1 -- 172.21.15.62:0/1235747769 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8017070 con 0x7fbde8067810 2024-04-18T10:06:22.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.654+0000 7fbdde7fc640 1 -- 172.21.15.62:0/1235747769 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdd8005cf0 con 0x7fbde8067810 2024-04-18T10:06:22.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.654+0000 7fbdefaa5640 1 -- 172.21.15.62:0/1235747769 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde81338e0 con 0x7fbde8067810 2024-04-18T10:06:22.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.654+0000 7fbdde7fc640 1 -- 172.21.15.62:0/1235747769 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd80072f0 con 0x7fbde8067810 2024-04-18T10:06:22.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.654+0000 7fbdefaa5640 1 -- 172.21.15.62:0/1235747769 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde8132a40 con 0x7fbde8067810 2024-04-18T10:06:22.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.654+0000 7fbdefaa5640 1 -- 172.21.15.62:0/1235747769 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde8069230 con 0x7fbde8067810 2024-04-18T10:06:22.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.654+0000 7fbdde7fc640 1 -- 172.21.15.62:0/1235747769 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbdd801a070 con 0x7fbde8067810 2024-04-18T10:06:22.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.654+0000 7fbdde7fc640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbdcc068380 0x7fbdcc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.654+0000 7fbdde7fc640 1 -- 172.21.15.62:0/1235747769 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbdd8085d10 con 0x7fbde8067810 2024-04-18T10:06:22.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.654+0000 7fbded040640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbdcc068380 0x7fbdcc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.654+0000 7fbded040640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbdcc068380 0x7fbdcc06a840 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbde80b98d0 tx=0x7fbde4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:22.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.658+0000 7fbdde7fc640 1 -- 172.21.15.62:0/1235747769 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbde81338e0 con 0x7fbde8067810 2024-04-18T10:06:22.813 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.814+0000 7fbdefaa5640 1 -- 172.21.15.62:0/1235747769 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbde809ea90 con 0x7fbde8067810 2024-04-18T10:06:22.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.814+0000 7fbdde7fc640 1 -- 172.21.15.62:0/1235747769 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fbdd8056360 con 0x7fbde8067810 2024-04-18T10:06:22.814 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-18T10:06:22.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.818+0000 7fbdefaa5640 1 -- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbdcc068380 msgr2=0x7fbdcc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.818+0000 7fbdefaa5640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbdcc068380 0x7fbdcc06a840 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbde80b98d0 tx=0x7fbde4008040 comp rx=0 tx=0).stop 2024-04-18T10:06:22.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.818+0000 7fbdefaa5640 1 -- 172.21.15.62:0/1235747769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbde8067810 msgr2=0x7fbde80b9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.818+0000 7fbdefaa5640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbde8067810 0x7fbde80b9210 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800c420 tx=0x7fbdd800c8e0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.818+0000 7fbdefaa5640 1 -- 172.21.15.62:0/1235747769 shutdown_connections 2024-04-18T10:06:22.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.818+0000 7fbdefaa5640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbdcc068380 0x7fbdcc06a840 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.818+0000 7fbdefaa5640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbde8146730 0x7fbde8139190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.818+0000 7fbdefaa5640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbde8067bf0 0x7fbde8133da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.818+0000 7fbdefaa5640 1 --2- 172.21.15.62:0/1235747769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbde8067810 0x7fbde80b9210 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.818+0000 7fbdefaa5640 1 -- 172.21.15.62:0/1235747769 >> 172.21.15.62:0/1235747769 conn(0x7fbde80ba870 msgr2=0x7fbde80b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:22.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.818+0000 7fbdefaa5640 1 -- 172.21.15.62:0/1235747769 shutdown_connections 2024-04-18T10:06:22.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.818+0000 7fbdefaa5640 1 -- 172.21.15.62:0/1235747769 wait complete. 2024-04-18T10:06:22.832 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-18T10:06:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f22468d1640 1 Processor -- start 2024-04-18T10:06:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f22468d1640 1 -- start start 2024-04-18T10:06:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f22468d1640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f22400b4c90 0x7f2240145f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f22468d1640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f224006e490 0x7f22401464c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f22468d1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f224009e090 0x7f224009e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f22468d1640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f224006d2a0 con 0x7f224006e490 2024-04-18T10:06:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f22468d1640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f224009ea40 con 0x7f22400b4c90 2024-04-18T10:06:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f22468d1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f224006d4f0 con 0x7f224009e090 2024-04-18T10:06:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f223f7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f224006e490 0x7f22401464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f223f7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f224006e490 0x7f22401464c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37630/0 (socket says 172.21.15.62:37630) 2024-04-18T10:06:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f223f7fe640 1 -- 172.21.15.62:0/3187485163 learned_addr learned my addr 172.21.15.62:0/3187485163 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f223f7fe640 1 -- 172.21.15.62:0/3187485163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f224009e090 msgr2=0x7f224009e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f2244e6e640 1 --2- 172.21.15.62:0/3187485163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f224009e090 0x7f224009e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f223ffff640 1 --2- 172.21.15.62:0/3187485163 >> v2:172.21.15.145:3300/0 conn(0x7f22400b4c90 0x7f2240145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f223f7fe640 1 --2- 172.21.15.62:0/3187485163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f224009e090 0x7f224009e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f223f7fe640 1 -- 172.21.15.62:0/3187485163 >> v2:172.21.15.145:3300/0 conn(0x7f22400b4c90 msgr2=0x7f2240145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f223f7fe640 1 --2- 172.21.15.62:0/3187485163 >> v2:172.21.15.145:3300/0 conn(0x7f22400b4c90 0x7f2240145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f223f7fe640 1 -- 172.21.15.62:0/3187485163 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f224006d7d0 con 0x7f224006e490 2024-04-18T10:06:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f223ffff640 1 --2- 172.21.15.62:0/3187485163 >> v2:172.21.15.145:3300/0 conn(0x7f22400b4c90 0x7f2240145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.950+0000 7f2244e6e640 1 --2- 172.21.15.62:0/3187485163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f224009e090 0x7f224009e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223f7fe640 1 --2- 172.21.15.62:0/3187485163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224006e490 0x7f22401464c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f224009bff0 tx=0x7f223401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=213fd4dcc66102a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223effd640 1 -- 172.21.15.62:0/3187485163 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22340b2070 con 0x7f224006e490 2024-04-18T10:06:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223effd640 1 -- 172.21.15.62:0/3187485163 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22340a9070 con 0x7f224006e490 2024-04-18T10:06:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223effd640 1 -- 172.21.15.62:0/3187485163 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22340ad880 con 0x7f224006e490 2024-04-18T10:06:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 -- 172.21.15.62:0/3187485163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224006e490 msgr2=0x7f22401464c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 --2- 172.21.15.62:0/3187485163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224006e490 0x7f22401464c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f224009bff0 tx=0x7f223401c490 comp rx=0 tx=0).stop 2024-04-18T10:06:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 -- 172.21.15.62:0/3187485163 shutdown_connections 2024-04-18T10:06:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 --2- 172.21.15.62:0/3187485163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f224009e090 0x7f224009e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 --2- 172.21.15.62:0/3187485163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224006e490 0x7f22401464c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 --2- 172.21.15.62:0/3187485163 >> v2:172.21.15.145:3300/0 conn(0x7f22400b4c90 0x7f2240145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 -- 172.21.15.62:0/3187485163 >> 172.21.15.62:0/3187485163 conn(0x7f22400b9690 msgr2=0x7f22400b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 -- 172.21.15.62:0/3187485163 shutdown_connections 2024-04-18T10:06:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 -- 172.21.15.62:0/3187485163 wait complete. 2024-04-18T10:06:22.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 Processor -- start 2024-04-18T10:06:22.955 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 -- start start 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f224006e490 0x7f224013ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224009e090 0x7f22400af260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22400b4c90 0x7f224014e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2240149a30 con 0x7f224009e090 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f22401498b0 con 0x7f22400b4c90 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f22468d1640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2240149bb0 con 0x7f224006e490 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224009e090 0x7f22400af260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224009e090 0x7f22400af260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37638/0 (socket says 172.21.15.62:37638) 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223f7fe640 1 -- 172.21.15.62:0/3345955008 learned_addr learned my addr 172.21.15.62:0/3345955008 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f2244e6e640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22400b4c90 0x7f224014e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223ffff640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f224006e490 0x7f224013ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223f7fe640 1 -- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f224006e490 msgr2=0x7f224013ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223f7fe640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f224006e490 0x7f224013ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223f7fe640 1 -- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22400b4c90 msgr2=0x7f224014e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:22.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223f7fe640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22400b4c90 0x7f224014e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223f7fe640 1 -- 172.21.15.62:0/3345955008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22400b9b10 con 0x7f224009e090 2024-04-18T10:06:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.954+0000 7f223f7fe640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224009e090 0x7f22400af260 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f223401ab40 tx=0x7f2234019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.958+0000 7f223cff9640 1 -- 172.21.15.62:0/3345955008 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22340b2040 con 0x7f224009e090 2024-04-18T10:06:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.958+0000 7f223cff9640 1 -- 172.21.15.62:0/3345955008 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22340b6620 con 0x7f224009e090 2024-04-18T10:06:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.958+0000 7f22468d1640 1 -- 172.21.15.62:0/3345955008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22400af7a0 con 0x7f224009e090 2024-04-18T10:06:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.958+0000 7f223cff9640 1 -- 172.21.15.62:0/3345955008 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22340b6a10 con 0x7f224009e090 2024-04-18T10:06:22.957 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.958+0000 7f22468d1640 1 -- 172.21.15.62:0/3345955008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22400b03f0 con 0x7f224009e090 2024-04-18T10:06:22.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.958+0000 7f22468d1640 1 -- 172.21.15.62:0/3345955008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22400ae5e0 con 0x7f224009e090 2024-04-18T10:06:22.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.958+0000 7f223cff9640 1 -- 172.21.15.62:0/3345955008 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f22340ab070 con 0x7f224009e090 2024-04-18T10:06:22.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.958+0000 7f223cff9640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f222c0683b0 0x7f222c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:22.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.958+0000 7f223cff9640 1 -- 172.21.15.62:0/3345955008 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f22341205d0 con 0x7f224009e090 2024-04-18T10:06:22.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.958+0000 7f223ffff640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f222c0683b0 0x7f222c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:22.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.958+0000 7f223ffff640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f222c0683b0 0x7f222c06a870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2230006fd0 tx=0x7f2230008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:22.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:22.962+0000 7f223cff9640 1 -- 172.21.15.62:0/3345955008 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f22400ae5e0 con 0x7f224009e090 2024-04-18T10:06:23.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.126+0000 7f22468d1640 1 -- 172.21.15.62:0/3345955008 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2240068c00 con 0x7f224009e090 2024-04-18T10:06:23.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.126+0000 7f223cff9640 1 -- 172.21.15.62:0/3345955008 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f2234016110 con 0x7f224009e090 2024-04-18T10:06:23.124 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-04-18T10:06:23.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.126+0000 7f22468d1640 1 -- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f222c0683b0 msgr2=0x7f222c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.126+0000 7f22468d1640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f222c0683b0 0x7f222c06a870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2230006fd0 tx=0x7f2230008040 comp rx=0 tx=0).stop 2024-04-18T10:06:23.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.126+0000 7f22468d1640 1 -- 172.21.15.62:0/3345955008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224009e090 msgr2=0x7f22400af260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.126+0000 7f22468d1640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224009e090 0x7f22400af260 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f223401ab40 tx=0x7f2234019230 comp rx=0 tx=0).stop 2024-04-18T10:06:23.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.130+0000 7f22468d1640 1 -- 172.21.15.62:0/3345955008 shutdown_connections 2024-04-18T10:06:23.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.130+0000 7f22468d1640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f222c0683b0 0x7f222c06a870 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.130+0000 7f22468d1640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22400b4c90 0x7f224014e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.130+0000 7f22468d1640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f224009e090 0x7f22400af260 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.130+0000 7f22468d1640 1 --2- 172.21.15.62:0/3345955008 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f224006e490 0x7f224013ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.130+0000 7f22468d1640 1 -- 172.21.15.62:0/3345955008 >> 172.21.15.62:0/3345955008 conn(0x7f22400b9690 msgr2=0x7f2240144f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:23.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.130+0000 7f22468d1640 1 -- 172.21.15.62:0/3345955008 shutdown_connections 2024-04-18T10:06:23.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.130+0000 7f22468d1640 1 -- 172.21.15.62:0/3345955008 wait complete. 2024-04-18T10:06:23.144 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-18T10:06:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221fb41640 1 Processor -- start 2024-04-18T10:06:23.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221fb41640 1 -- start start 2024-04-18T10:06:23.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221fb41640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f221809c4d0 0x7f221809ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221fb41640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f221809e2d0 0x7f2218142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221fb41640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f221809d430 con 0x7f221809b620 2024-04-18T10:06:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221fb41640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f221806eb30 con 0x7f221809e2d0 2024-04-18T10:06:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221fb41640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f221809dea0 con 0x7f221809c4d0 2024-04-18T10:06:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221e0de640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f221809e2d0 0x7f2218142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221d8dd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f221809b620 0x7f22181492b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35296/0 (socket says 172.21.15.62:35296) 2024-04-18T10:06:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221d8dd640 1 -- 172.21.15.62:0/153956127 learned_addr learned my addr 172.21.15.62:0/153956127 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221d0dc640 1 --2- 172.21.15.62:0/153956127 >> v2:172.21.15.145:3300/0 conn(0x7f221809c4d0 0x7f221809ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221e0de640 1 -- 172.21.15.62:0/153956127 >> v2:172.21.15.145:3300/0 conn(0x7f221809c4d0 msgr2=0x7f221809ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221e0de640 1 --2- 172.21.15.62:0/153956127 >> v2:172.21.15.145:3300/0 conn(0x7f221809c4d0 0x7f221809ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221e0de640 1 -- 172.21.15.62:0/153956127 >> v1:172.21.15.62:6789/0 conn(0x7f221809b620 legacy=0x7f22181492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.270+0000 7f221e0de640 1 -- 172.21.15.62:0/153956127 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f221809d6a0 con 0x7f221809e2d0 2024-04-18T10:06:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221e0de640 1 --2- 172.21.15.62:0/153956127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f221809e2d0 0x7f2218142ba0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f22100097c0 tx=0x7f221001c110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c041582d053d19 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221c8db640 1 -- 172.21.15.62:0/153956127 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22100b0070 con 0x7f221809e2d0 2024-04-18T10:06:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221c8db640 1 -- 172.21.15.62:0/153956127 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22100ab600 con 0x7f221809e2d0 2024-04-18T10:06:23.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221c8db640 1 -- 172.21.15.62:0/153956127 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22100ab9d0 con 0x7f221809e2d0 2024-04-18T10:06:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221fb41640 1 -- 172.21.15.62:0/153956127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f221809e2d0 msgr2=0x7f2218142ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221fb41640 1 --2- 172.21.15.62:0/153956127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f221809e2d0 0x7f2218142ba0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f22100097c0 tx=0x7f221001c110 comp rx=0 tx=0).stop 2024-04-18T10:06:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221fb41640 1 -- 172.21.15.62:0/153956127 shutdown_connections 2024-04-18T10:06:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221fb41640 1 --2- 172.21.15.62:0/153956127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f221809e2d0 0x7f2218142ba0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221fb41640 1 --2- 172.21.15.62:0/153956127 >> v2:172.21.15.145:3300/0 conn(0x7f221809c4d0 0x7f221809ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221fb41640 1 -- 172.21.15.62:0/153956127 >> 172.21.15.62:0/153956127 conn(0x7f22180b9420 msgr2=0x7f22180b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221fb41640 1 -- 172.21.15.62:0/153956127 shutdown_connections 2024-04-18T10:06:23.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221fb41640 1 -- 172.21.15.62:0/153956127 wait complete. 2024-04-18T10:06:23.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221fb41640 1 Processor -- start 2024-04-18T10:06:23.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.274+0000 7f221fb41640 1 -- start start 2024-04-18T10:06:23.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221fb41640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f221809b620 0x7f22180ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221fb41640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f221809c4d0 0x7f22181de750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221fb41640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f221809e2d0 0x7f22181dec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221fb41640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2218152b90 con 0x7f221809b620 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221fb41640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f2218152a10 con 0x7f221809e2d0 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221fb41640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2218152d10 con 0x7f221809c4d0 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221d8dd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f221809b620 0x7f22180ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221d8dd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f221809b620 0x7f22180ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37646/0 (socket says 172.21.15.62:37646) 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221d8dd640 1 -- 172.21.15.62:0/2411334807 learned_addr learned my addr 172.21.15.62:0/2411334807 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221d8dd640 1 -- 172.21.15.62:0/2411334807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f221809c4d0 msgr2=0x7f22181de750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221d8dd640 1 --2- 172.21.15.62:0/2411334807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f221809c4d0 0x7f22181de750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221d8dd640 1 -- 172.21.15.62:0/2411334807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f221809e2d0 msgr2=0x7f22181dec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221d8dd640 1 --2- 172.21.15.62:0/2411334807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f221809e2d0 0x7f22181dec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221d8dd640 1 -- 172.21.15.62:0/2411334807 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22180b14d0 con 0x7f221809b620 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221d8dd640 1 --2- 172.21.15.62:0/2411334807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f221809b620 0x7f22180ae7e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f220800cb10 tx=0x7f220800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f22167fc640 1 -- 172.21.15.62:0/2411334807 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2208017070 con 0x7f221809b620 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f22167fc640 1 -- 172.21.15.62:0/2411334807 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f220800ac70 con 0x7f221809b620 2024-04-18T10:06:23.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f22167fc640 1 -- 172.21.15.62:0/2411334807 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f220800ae60 con 0x7f221809b620 2024-04-18T10:06:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221fb41640 1 -- 172.21.15.62:0/2411334807 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22180af230 con 0x7f221809b620 2024-04-18T10:06:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221fb41640 1 -- 172.21.15.62:0/2411334807 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22180b11f0 con 0x7f221809b620 2024-04-18T10:06:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f221fb41640 1 -- 172.21.15.62:0/2411334807 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2218068b20 con 0x7f221809b620 2024-04-18T10:06:23.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f22167fc640 1 -- 172.21.15.62:0/2411334807 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f220801a070 con 0x7f221809b620 2024-04-18T10:06:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f22167fc640 1 --2- 172.21.15.62:0/2411334807 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2204068380 0x7f220406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.278+0000 7f22167fc640 1 -- 172.21.15.62:0/2411334807 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f22080867c0 con 0x7f221809b620 2024-04-18T10:06:23.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.282+0000 7f221d0dc640 1 --2- 172.21.15.62:0/2411334807 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2204068380 0x7f220406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.282+0000 7f221d0dc640 1 --2- 172.21.15.62:0/2411334807 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2204068380 0x7f220406a840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f221809bbe0 tx=0x7f220c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:23.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.286+0000 7f22167fc640 1 -- 172.21.15.62:0/2411334807 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2208056d80 con 0x7f221809b620 2024-04-18T10:06:23.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.466+0000 7f221fb41640 1 -- 172.21.15.62:0/2411334807 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f221806e2b0 con 0x7f221809b620 2024-04-18T10:06:23.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.466+0000 7f22167fc640 1 -- 172.21.15.62:0/2411334807 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f2208056680 con 0x7f221809b620 2024-04-18T10:06:23.465 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-04-18T10:06:23.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.470+0000 7f221fb41640 1 -- 172.21.15.62:0/2411334807 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2204068380 msgr2=0x7f220406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.470+0000 7f221fb41640 1 --2- 172.21.15.62:0/2411334807 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2204068380 0x7f220406a840 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f221809bbe0 tx=0x7f220c008040 comp rx=0 tx=0).stop 2024-04-18T10:06:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.470+0000 7f221fb41640 1 -- 172.21.15.62:0/2411334807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f221809b620 msgr2=0x7f22180ae7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.470+0000 7f221fb41640 1 --2- 172.21.15.62:0/2411334807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f221809b620 0x7f22180ae7e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f220800cb10 tx=0x7f220800a060 comp rx=0 tx=0).stop 2024-04-18T10:06:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.470+0000 7f221fb41640 1 -- 172.21.15.62:0/2411334807 shutdown_connections 2024-04-18T10:06:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.470+0000 7f221fb41640 1 --2- 172.21.15.62:0/2411334807 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2204068380 0x7f220406a840 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.470+0000 7f221fb41640 1 --2- 172.21.15.62:0/2411334807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f221809e2d0 0x7f22181dec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.470+0000 7f221fb41640 1 --2- 172.21.15.62:0/2411334807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f221809c4d0 0x7f22181de750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.470+0000 7f221fb41640 1 --2- 172.21.15.62:0/2411334807 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f221809b620 0x7f22180ae7e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.470+0000 7f221fb41640 1 -- 172.21.15.62:0/2411334807 >> 172.21.15.62:0/2411334807 conn(0x7f22180b9420 msgr2=0x7f2218073830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.470+0000 7f221fb41640 1 -- 172.21.15.62:0/2411334807 shutdown_connections 2024-04-18T10:06:23.468 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.470+0000 7f221fb41640 1 -- 172.21.15.62:0/2411334807 wait complete. 2024-04-18T10:06:23.484 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:06:23.485 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-18T10:06:23.490 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddfd46640 1 Processor -- start 2024-04-18T10:06:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddfd46640 1 -- start start 2024-04-18T10:06:23.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddfd46640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd80b9d90 0x7f5dd81450e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddfd46640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dd8145620 con 0x7f5dd80b44e0 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddfd46640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dd8148f50 con 0x7f5dd80b4940 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddfd46640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd80b4dd0 con 0x7f5dd80b9d90 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddd2e1640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5dd80b44e0 0x7f5dd8148130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35304/0 (socket says 172.21.15.62:35304) 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddd2e1640 1 -- 172.21.15.62:0/2814535912 learned_addr learned my addr 172.21.15.62:0/2814535912 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5dddae2640 1 --2- 172.21.15.62:0/2814535912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd80b9d90 0x7f5dd81450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4238736391 0 0) 0x7f5dc8001120 con 0x7f5dd80b44e0 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5dc4003530 con 0x7f5dd80b44e0 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 364157401 0 0) 0x7f5dc4003530 con 0x7f5dd80b44e0 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd80b9d90 msgr2=0x7f5dd81450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 --2- 172.21.15.62:0/2814535912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd80b9d90 0x7f5dd81450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 >> v1:172.21.15.145:6789/0 conn(0x7f5dd80b4940 legacy=0x7f5dd8148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd80ba170 con 0x7f5dd80b44e0 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f5dc8001850 con 0x7f5dd80b44e0 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 >> v1:172.21.15.62:6789/0 conn(0x7f5dd80b44e0 legacy=0x7f5dd8148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 --2- 172.21.15.62:0/2814535912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dc40039f0 0x7f5dc4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 --2- 172.21.15.62:0/2814535912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd80b9d90 0x7f5dc4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 --2- 172.21.15.62:0/2814535912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dc4004d50 0x7f5dc40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc400dce0 con 0x7f5dc4004d50 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc400de90 con 0x7f5dc40039f0 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f5dc400e090 con 0x7f5dd80b9d90 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5dde2e3640 1 --2- 172.21.15.62:0/2814535912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dc4004d50 0x7f5dc40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5ddd2e1640 1 --2- 172.21.15.62:0/2814535912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd80b9d90 0x7f5dc4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5dddae2640 1 --2- 172.21.15.62:0/2814535912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dc40039f0 0x7f5dc4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5dde2e3640 1 -- 172.21.15.62:0/2814535912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd80b9d90 msgr2=0x7f5dc4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5dde2e3640 1 --2- 172.21.15.62:0/2814535912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd80b9d90 0x7f5dc4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5dde2e3640 1 -- 172.21.15.62:0/2814535912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dc40039f0 msgr2=0x7f5dc4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:23.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5dde2e3640 1 --2- 172.21.15.62:0/2814535912 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dc40039f0 0x7f5dc4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.654+0000 7f5dde2e3640 1 -- 172.21.15.62:0/2814535912 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc4003530 con 0x7f5dc4004d50 2024-04-18T10:06:23.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dde2e3640 1 --2- 172.21.15.62:0/2814535912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dc4004d50 0x7f5dc40051d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0001e70 tx=0x7f5dd001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52544fa2f158b09 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:23.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0001a90 con 0x7f5dc4004d50 2024-04-18T10:06:23.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd0003b80 con 0x7f5dc4004d50 2024-04-18T10:06:23.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddcae0640 1 -- 172.21.15.62:0/2814535912 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0005080 con 0x7f5dc4004d50 2024-04-18T10:06:23.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 -- 172.21.15.62:0/2814535912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dc4004d50 msgr2=0x7f5dc40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 --2- 172.21.15.62:0/2814535912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dc4004d50 0x7f5dc40051d0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0001e70 tx=0x7f5dd001c780 comp rx=0 tx=0).stop 2024-04-18T10:06:23.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dde2e3640 1 -- 172.21.15.62:0/2814535912 reap_dead start 2024-04-18T10:06:23.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 -- 172.21.15.62:0/2814535912 shutdown_connections 2024-04-18T10:06:23.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 -- 172.21.15.62:0/2814535912 >> 172.21.15.62:0/2814535912 conn(0x7f5dd806c0d0 msgr2=0x7f5dd80b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:23.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 -- 172.21.15.62:0/2814535912 shutdown_connections 2024-04-18T10:06:23.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 -- 172.21.15.62:0/2814535912 wait complete. 2024-04-18T10:06:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 Processor -- start 2024-04-18T10:06:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 -- start start 2024-04-18T10:06:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dc4004d50 0x7f5dd80b27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd8143300 0x7f5dd8143780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dd808d530 0x7f5dd808d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd80b3fc0 con 0x7f5dc4004d50 2024-04-18T10:06:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd8145b80 con 0x7f5dd808d530 2024-04-18T10:06:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd80b5b20 con 0x7f5dd8143300 2024-04-18T10:06:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dddae2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dc4004d50 0x7f5dd80b27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dddae2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dc4004d50 0x7f5dd80b27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37656/0 (socket says 172.21.15.62:37656) 2024-04-18T10:06:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dddae2640 1 -- 172.21.15.62:0/605959450 learned_addr learned my addr 172.21.15.62:0/605959450 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:23.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddd2e1640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd8143300 0x7f5dd8143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dde2e3640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dd808d530 0x7f5dd808d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dddae2640 1 -- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd8143300 msgr2=0x7f5dd8143780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dddae2640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd8143300 0x7f5dd8143780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dddae2640 1 -- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dd808d530 msgr2=0x7f5dd808d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dddae2640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dd808d530 0x7f5dd808d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dddae2640 1 -- 172.21.15.62:0/605959450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd808def0 con 0x7f5dc4004d50 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dde2e3640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dd808d530 0x7f5dd808d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dddae2640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dc4004d50 0x7f5dd80b27b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40115e0 tx=0x7f5dd4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dce7fc640 1 -- 172.21.15.62:0/605959450 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4015870 con 0x7f5dc4004d50 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dce7fc640 1 -- 172.21.15.62:0/605959450 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dd4015a10 con 0x7f5dc4004d50 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5dce7fc640 1 -- 172.21.15.62:0/605959450 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4015dc0 con 0x7f5dc4004d50 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.658+0000 7f5ddfd46640 1 -- 172.21.15.62:0/605959450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd808e9e0 con 0x7f5dc4004d50 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.662+0000 7f5ddfd46640 1 -- 172.21.15.62:0/605959450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd808ece0 con 0x7f5dc4004d50 2024-04-18T10:06:23.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.662+0000 7f5ddfd46640 1 -- 172.21.15.62:0/605959450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd8068b90 con 0x7f5dc4004d50 2024-04-18T10:06:23.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.662+0000 7f5dce7fc640 1 -- 172.21.15.62:0/605959450 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5dd40143e0 con 0x7f5dc4004d50 2024-04-18T10:06:23.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.662+0000 7f5dce7fc640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5dbc0683b0 0x7f5dbc06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.662+0000 7f5dce7fc640 1 -- 172.21.15.62:0/605959450 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5dd40871f0 con 0x7f5dc4004d50 2024-04-18T10:06:23.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.662+0000 7f5ddd2e1640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5dbc0683b0 0x7f5dbc06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.662+0000 7f5ddd2e1640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5dbc0683b0 0x7f5dbc06a870 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8144890 tx=0x7f5dc8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:23.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.666+0000 7f5dce7fc640 1 -- 172.21.15.62:0/605959450 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5dd808def0 con 0x7f5dc4004d50 2024-04-18T10:06:23.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.826+0000 7f5ddfd46640 1 -- 172.21.15.62:0/605959450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5dd808ff70 con 0x7f5dc4004d50 2024-04-18T10:06:23.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5dce7fc640 1 -- 172.21.15.62:0/605959450 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f5dd4019030 con 0x7f5dc4004d50 2024-04-18T10:06:23.828 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:23.828 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:23.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5ddfd46640 1 -- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5dbc0683b0 msgr2=0x7f5dbc06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5ddfd46640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5dbc0683b0 0x7f5dbc06a870 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8144890 tx=0x7f5dc8009040 comp rx=0 tx=0).stop 2024-04-18T10:06:23.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5ddfd46640 1 -- 172.21.15.62:0/605959450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dc4004d50 msgr2=0x7f5dd80b27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5ddfd46640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dc4004d50 0x7f5dd80b27b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5dd40115e0 tx=0x7f5dd4011aa0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5ddfd46640 1 -- 172.21.15.62:0/605959450 shutdown_connections 2024-04-18T10:06:23.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5ddfd46640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5dbc0683b0 0x7f5dbc06a870 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5ddfd46640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5dd808d530 0x7f5dd808d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5ddfd46640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5dd8143300 0x7f5dd8143780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5ddfd46640 1 --2- 172.21.15.62:0/605959450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5dc4004d50 0x7f5dd80b27b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5ddfd46640 1 -- 172.21.15.62:0/605959450 >> 172.21.15.62:0/605959450 conn(0x7f5dd806c0d0 msgr2=0x7f5dd80b6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:23.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5ddfd46640 1 -- 172.21.15.62:0/605959450 shutdown_connections 2024-04-18T10:06:23.831 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.830+0000 7f5ddfd46640 1 -- 172.21.15.62:0/605959450 wait complete. 2024-04-18T10:06:23.847 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-18T10:05:56.782649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-18T10:05:57.541623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:23.847 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:23.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252f3e4640 1 Processor -- start 2024-04-18T10:06:23.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252f3e4640 1 -- start start 2024-04-18T10:06:23.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252f3e4640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f252806d3a0 0x7f252809c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252f3e4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25280b4530 0x7f25280b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252f3e4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f25280b9960 0x7f2528141d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252f3e4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f252809b980 con 0x7f25280b9960 2024-04-18T10:06:23.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252f3e4640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f25280b35d0 con 0x7f252806d3a0 2024-04-18T10:06:23.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252f3e4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f252806d780 con 0x7f25280b4530 2024-04-18T10:06:23.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252d981640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f25280b9960 0x7f2528141d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252d981640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f25280b9960 0x7f2528141d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37658/0 (socket says 172.21.15.62:37658) 2024-04-18T10:06:23.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252d981640 1 -- 172.21.15.62:0/2080069288 learned_addr learned my addr 172.21.15.62:0/2080069288 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:23.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252d180640 1 --2- 172.21.15.62:0/2080069288 >> v2:172.21.15.145:3300/0 conn(0x7f252806d3a0 0x7f252809c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252c97f640 1 --2- 172.21.15.62:0/2080069288 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25280b4530 0x7f25280b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252d981640 1 -- 172.21.15.62:0/2080069288 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25280b4530 msgr2=0x7f25280b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:23.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252d981640 1 --2- 172.21.15.62:0/2080069288 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25280b4530 0x7f25280b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252d981640 1 -- 172.21.15.62:0/2080069288 >> v2:172.21.15.145:3300/0 conn(0x7f252806d3a0 msgr2=0x7f252809c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:23.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252d981640 1 --2- 172.21.15.62:0/2080069288 >> v2:172.21.15.145:3300/0 conn(0x7f252806d3a0 0x7f252809c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252d981640 1 -- 172.21.15.62:0/2080069288 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252809af10 con 0x7f25280b9960 2024-04-18T10:06:23.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.966+0000 7f252d981640 1 --2- 172.21.15.62:0/2080069288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25280b9960 0x7f2528141d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f25280bae60 tx=0x7f252001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63707b364d91edda server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:23.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f251ffff640 1 -- 172.21.15.62:0/2080069288 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25200b4070 con 0x7f25280b9960 2024-04-18T10:06:23.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f251ffff640 1 -- 172.21.15.62:0/2080069288 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25200af400 con 0x7f25280b9960 2024-04-18T10:06:23.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f251ffff640 1 -- 172.21.15.62:0/2080069288 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25200af7d0 con 0x7f25280b9960 2024-04-18T10:06:23.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 -- 172.21.15.62:0/2080069288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25280b9960 msgr2=0x7f2528141d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 --2- 172.21.15.62:0/2080069288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25280b9960 0x7f2528141d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f25280bae60 tx=0x7f252001f1e0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 -- 172.21.15.62:0/2080069288 shutdown_connections 2024-04-18T10:06:23.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 --2- 172.21.15.62:0/2080069288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25280b9960 0x7f2528141d50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 --2- 172.21.15.62:0/2080069288 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25280b4530 0x7f25280b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 --2- 172.21.15.62:0/2080069288 >> v2:172.21.15.145:3300/0 conn(0x7f252806d3a0 0x7f252809c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 -- 172.21.15.62:0/2080069288 >> 172.21.15.62:0/2080069288 conn(0x7f25280ba0c0 msgr2=0x7f25280ba4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:23.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 -- 172.21.15.62:0/2080069288 shutdown_connections 2024-04-18T10:06:23.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 -- 172.21.15.62:0/2080069288 wait complete. 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 Processor -- start 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 -- start start 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f252806d3a0 0x7f25281d3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25280b4530 0x7f25281e0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25280b9960 0x7f2528147ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2528142290 con 0x7f25280b9960 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f252809bee0 con 0x7f25280b4530 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2528142410 con 0x7f252806d3a0 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252d981640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25280b9960 0x7f2528147ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252d981640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25280b9960 0x7f2528147ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37670/0 (socket says 172.21.15.62:37670) 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252d981640 1 -- 172.21.15.62:0/2701526070 learned_addr learned my addr 172.21.15.62:0/2701526070 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252d180640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f252806d3a0 0x7f25281d3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252c97f640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25280b4530 0x7f25281e0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252d981640 1 -- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f252806d3a0 msgr2=0x7f25281d3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252d981640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f252806d3a0 0x7f25281d3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252d981640 1 -- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25280b4530 msgr2=0x7f25281e0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252d981640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25280b4530 0x7f25281e0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252d981640 1 -- 172.21.15.62:0/2701526070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2528140760 con 0x7f25280b9960 2024-04-18T10:06:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252d981640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25280b9960 0x7f2528147ef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f25200b2040 tx=0x7f252001bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f251dffb640 1 -- 172.21.15.62:0/2701526070 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25200b4040 con 0x7f25280b9960 2024-04-18T10:06:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f251dffb640 1 -- 172.21.15.62:0/2701526070 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25200afce0 con 0x7f25280b9960 2024-04-18T10:06:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 -- 172.21.15.62:0/2701526070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25281d3e20 con 0x7f25280b9960 2024-04-18T10:06:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f251dffb640 1 -- 172.21.15.62:0/2701526070 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520009510 con 0x7f25280b9960 2024-04-18T10:06:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.970+0000 7f252f3e4640 1 -- 172.21.15.62:0/2701526070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2528148ac0 con 0x7f25280b9960 2024-04-18T10:06:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.974+0000 7f252f3e4640 1 -- 172.21.15.62:0/2701526070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2528069230 con 0x7f25280b9960 2024-04-18T10:06:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.974+0000 7f251dffb640 1 -- 172.21.15.62:0/2701526070 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f25200c3020 con 0x7f25280b9960 2024-04-18T10:06:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.974+0000 7f251dffb640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f250c068380 0x7f250c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.974+0000 7f251dffb640 1 -- 172.21.15.62:0/2701526070 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2520124350 con 0x7f25280b9960 2024-04-18T10:06:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.974+0000 7f252d180640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f250c068380 0x7f250c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.974+0000 7f252d180640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f250c068380 0x7f250c06a840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2524009af0 tx=0x7f2524008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:23.978+0000 7f251dffb640 1 -- 172.21.15.62:0/2701526070 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f25200ca020 con 0x7f25280b9960 2024-04-18T10:06:24.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.134+0000 7f252f3e4640 1 -- 172.21.15.62:0/2701526070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f25281d2de0 con 0x7f25280b9960 2024-04-18T10:06:24.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f251dffb640 1 -- 172.21.15.62:0/2701526070 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f2520018d40 con 0x7f25280b9960 2024-04-18T10:06:24.135 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f252f3e4640 1 -- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f250c068380 msgr2=0x7f250c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f252f3e4640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f250c068380 0x7f250c06a840 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2524009af0 tx=0x7f2524008040 comp rx=0 tx=0).stop 2024-04-18T10:06:24.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f252f3e4640 1 -- 172.21.15.62:0/2701526070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25280b9960 msgr2=0x7f2528147ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f252f3e4640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25280b9960 0x7f2528147ef0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f25200b2040 tx=0x7f252001bf80 comp rx=0 tx=0).stop 2024-04-18T10:06:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f252f3e4640 1 -- 172.21.15.62:0/2701526070 shutdown_connections 2024-04-18T10:06:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f252f3e4640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f250c068380 0x7f250c06a840 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f252f3e4640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f25280b9960 0x7f2528147ef0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f252f3e4640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25280b4530 0x7f25281e0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f252f3e4640 1 --2- 172.21.15.62:0/2701526070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f252806d3a0 0x7f25281d3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f252f3e4640 1 -- 172.21.15.62:0/2701526070 >> 172.21.15.62:0/2701526070 conn(0x7f25280ba0c0 msgr2=0x7f2528140050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f252f3e4640 1 -- 172.21.15.62:0/2701526070 shutdown_connections 2024-04-18T10:06:24.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.138+0000 7f252f3e4640 1 -- 172.21.15.62:0/2701526070 wait complete. 2024-04-18T10:06:24.153 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-18T10:06:24.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe1e049640 1 Processor -- start 2024-04-18T10:06:24.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe1e049640 1 -- start start 2024-04-18T10:06:24.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe1e049640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe180b01a0 0x7fbe1806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe1e049640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbe180ae4e0 0x7fbe1813e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe1e049640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe180b0580 con 0x7fbe180737c0 2024-04-18T10:06:24.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe1e049640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fbe180b59d0 con 0x7fbe180ae4e0 2024-04-18T10:06:24.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe1e049640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fbe180ae970 con 0x7fbe180b01a0 2024-04-18T10:06:24.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe17fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbe180737c0 0x7fbe18148100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35310/0 (socket says 172.21.15.62:35310) 2024-04-18T10:06:24.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe17fff640 1 -- 172.21.15.62:0/4268806448 learned_addr learned my addr 172.21.15.62:0/4268806448 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:24.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe177fe640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe180b01a0 0x7fbe1806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe16ffd640 1 --2- 172.21.15.62:0/4268806448 >> v2:172.21.15.145:3300/0 conn(0x7fbe180ae4e0 0x7fbe1813e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1882612636 0 0) 0x7fbe180b0580 con 0x7fbe180737c0 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe00003530 con 0x7fbe180737c0 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2085883919 0 0) 0x7fbe00003530 con 0x7fbe180737c0 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe180b01a0 msgr2=0x7fbe1806d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe180b01a0 0x7fbe1806d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 >> v2:172.21.15.145:3300/0 conn(0x7fbe180ae4e0 msgr2=0x7fbe1813e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 --2- 172.21.15.62:0/4268806448 >> v2:172.21.15.145:3300/0 conn(0x7fbe180ae4e0 0x7fbe1813e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe180b53a0 con 0x7fbe180737c0 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fbe10001a00 con 0x7fbe180737c0 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 >> v1:172.21.15.62:6789/0 conn(0x7fbe180737c0 legacy=0x7fbe18148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe000039c0 0x7fbe00003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00004810 0x7fbe00004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe180b01a0 0x7fbe000051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbe180b59d0 con 0x7fbe00004810 2024-04-18T10:06:24.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fbe180ae970 con 0x7fbe000039c0 2024-04-18T10:06:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fbe0000dc60 con 0x7fbe180b01a0 2024-04-18T10:06:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe16ffd640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00004810 0x7fbe00004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe177fe640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe000039c0 0x7fbe00003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe17fff640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe180b01a0 0x7fbe000051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe16ffd640 1 -- 172.21.15.62:0/4268806448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe180b01a0 msgr2=0x7fbe000051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe16ffd640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe180b01a0 0x7fbe000051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe16ffd640 1 -- 172.21.15.62:0/4268806448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe000039c0 msgr2=0x7fbe00003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe16ffd640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe000039c0 0x7fbe00003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe16ffd640 1 -- 172.21.15.62:0/4268806448 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe00003530 con 0x7fbe00004810 2024-04-18T10:06:24.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.274+0000 7fbe17fff640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe180b01a0 0x7fbe000051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:24.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe16ffd640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00004810 0x7fbe00004c90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009ae0 tx=0x7fbe0801cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0cebaac255625a server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:24.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080b2070 con 0x7fbe00004810 2024-04-18T10:06:24.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe080a5570 con 0x7fbe00004810 2024-04-18T10:06:24.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe167fc640 1 -- 172.21.15.62:0/4268806448 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080096b0 con 0x7fbe00004810 2024-04-18T10:06:24.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 -- 172.21.15.62:0/4268806448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00004810 msgr2=0x7fbe00004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 --2- 172.21.15.62:0/4268806448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe00004810 0x7fbe00004c90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009ae0 tx=0x7fbe0801cab0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe17fff640 1 -- 172.21.15.62:0/4268806448 reap_dead start 2024-04-18T10:06:24.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 -- 172.21.15.62:0/4268806448 shutdown_connections 2024-04-18T10:06:24.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 -- 172.21.15.62:0/4268806448 >> 172.21.15.62:0/4268806448 conn(0x7fbe180b9690 msgr2=0x7fbe180b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:24.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 -- 172.21.15.62:0/4268806448 shutdown_connections 2024-04-18T10:06:24.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 -- 172.21.15.62:0/4268806448 wait complete. 2024-04-18T10:06:24.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 Processor -- start 2024-04-18T10:06:24.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 -- start start 2024-04-18T10:06:24.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe00004810 0x7fbe1809daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe181e4960 0x7fbe180b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe180b7800 0x7fbe180b7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1806d6b0 con 0x7fbe181e4960 2024-04-18T10:06:24.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fbe1806e470 con 0x7fbe180b7800 2024-04-18T10:06:24.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe1e049640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fbe1806d830 con 0x7fbe00004810 2024-04-18T10:06:24.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe16ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe181e4960 0x7fbe180b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe16ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe181e4960 0x7fbe180b72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37676/0 (socket says 172.21.15.62:37676) 2024-04-18T10:06:24.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe16ffd640 1 -- 172.21.15.62:0/3187795115 learned_addr learned my addr 172.21.15.62:0/3187795115 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:24.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe17fff640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe180b7800 0x7fbe180b7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe177fe640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe00004810 0x7fbe1809daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe16ffd640 1 -- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe00004810 msgr2=0x7fbe1809daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe16ffd640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe00004810 0x7fbe1809daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe16ffd640 1 -- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe180b7800 msgr2=0x7fbe180b7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe16ffd640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe180b7800 0x7fbe180b7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe16ffd640 1 -- 172.21.15.62:0/3187795115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe18070dd0 con 0x7fbe181e4960 2024-04-18T10:06:24.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.278+0000 7fbe177fe640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe00004810 0x7fbe1809daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:24.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.282+0000 7fbe16ffd640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe181e4960 0x7fbe180b72c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbe080b0040 tx=0x7fbe00005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:24.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.282+0000 7fbdfffff640 1 -- 172.21.15.62:0/3187795115 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe080b2040 con 0x7fbe181e4960 2024-04-18T10:06:24.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.282+0000 7fbdfffff640 1 -- 172.21.15.62:0/3187795115 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe0800f000 con 0x7fbe181e4960 2024-04-18T10:06:24.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.282+0000 7fbe1e049640 1 -- 172.21.15.62:0/3187795115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe181e4160 con 0x7fbe181e4960 2024-04-18T10:06:24.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.282+0000 7fbdfffff640 1 -- 172.21.15.62:0/3187795115 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe08004020 con 0x7fbe181e4960 2024-04-18T10:06:24.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.282+0000 7fbe1e049640 1 -- 172.21.15.62:0/3187795115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe180b3900 con 0x7fbe181e4960 2024-04-18T10:06:24.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.282+0000 7fbe1e049640 1 -- 172.21.15.62:0/3187795115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe18068c00 con 0x7fbe181e4960 2024-04-18T10:06:24.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.282+0000 7fbdfffff640 1 -- 172.21.15.62:0/3187795115 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbe080c0030 con 0x7fbe181e4960 2024-04-18T10:06:24.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.282+0000 7fbdfffff640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbe04068380 0x7fbe0406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.282+0000 7fbdfffff640 1 -- 172.21.15.62:0/3187795115 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbe08120610 con 0x7fbe181e4960 2024-04-18T10:06:24.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.282+0000 7fbe177fe640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbe04068380 0x7fbe0406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.282+0000 7fbe177fe640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbe04068380 0x7fbe0406a840 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c006d50 tx=0x7fbe0c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:24.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.286+0000 7fbdfffff640 1 -- 172.21.15.62:0/3187795115 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbe080c7060 con 0x7fbe181e4960 2024-04-18T10:06:24.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.450+0000 7fbe1e049640 1 -- 172.21.15.62:0/3187795115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbe1813d1a0 con 0x7fbe181e4960 2024-04-18T10:06:24.449 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.450+0000 7fbdfffff640 1 -- 172.21.15.62:0/3187795115 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fbe080167c0 con 0x7fbe181e4960 2024-04-18T10:06:24.450 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-18T10:06:24.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.454+0000 7fbe1e049640 1 -- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbe04068380 msgr2=0x7fbe0406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.454+0000 7fbe1e049640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbe04068380 0x7fbe0406a840 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c006d50 tx=0x7fbe0c008040 comp rx=0 tx=0).stop 2024-04-18T10:06:24.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.454+0000 7fbe1e049640 1 -- 172.21.15.62:0/3187795115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe181e4960 msgr2=0x7fbe180b72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.454+0000 7fbe1e049640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe181e4960 0x7fbe180b72c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbe080b0040 tx=0x7fbe00005840 comp rx=0 tx=0).stop 2024-04-18T10:06:24.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.454+0000 7fbe1e049640 1 -- 172.21.15.62:0/3187795115 shutdown_connections 2024-04-18T10:06:24.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.454+0000 7fbe1e049640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbe04068380 0x7fbe0406a840 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.454+0000 7fbe1e049640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe180b7800 0x7fbe180b7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.454+0000 7fbe1e049640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbe181e4960 0x7fbe180b72c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.453 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.454+0000 7fbe1e049640 1 --2- 172.21.15.62:0/3187795115 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe00004810 0x7fbe1809daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.454+0000 7fbe1e049640 1 -- 172.21.15.62:0/3187795115 >> 172.21.15.62:0/3187795115 conn(0x7fbe180b9690 msgr2=0x7fbe1809b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:24.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.454+0000 7fbe1e049640 1 -- 172.21.15.62:0/3187795115 shutdown_connections 2024-04-18T10:06:24.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.454+0000 7fbe1e049640 1 -- 172.21.15.62:0/3187795115 wait complete. 2024-04-18T10:06:24.469 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-18T10:06:24.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80af8b4640 1 Processor -- start 2024-04-18T10:06:24.596 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80af8b4640 1 -- start start 2024-04-18T10:06:24.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80af8b4640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f80a80b64d0 0x7f80a814b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80af8b4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a806c110 0x7f80a814bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80af8b4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80a80b0de0 con 0x7f80a806bd30 2024-04-18T10:06:24.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80af8b4640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f80a80b17c0 con 0x7f80a80b64d0 2024-04-18T10:06:24.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80af8b4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f80a80b1570 con 0x7f80a806c110 2024-04-18T10:06:24.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80ace4f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f80a80b64d0 0x7f80a814b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80ace4f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f80a80b64d0 0x7f80a814b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37692/0 (socket says 172.21.15.62:37692) 2024-04-18T10:06:24.597 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80ace4f640 1 -- 172.21.15.62:0/2463102014 learned_addr learned my addr 172.21.15.62:0/2463102014 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:24.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80ade51640 1 --2- 172.21.15.62:0/2463102014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a806c110 0x7f80a814bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80ace4f640 1 -- 172.21.15.62:0/2463102014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a806c110 msgr2=0x7f80a814bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:24.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80ace4f640 1 --2- 172.21.15.62:0/2463102014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a806c110 0x7f80a814bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80ace4f640 1 -- 172.21.15.62:0/2463102014 >> v1:172.21.15.145:6789/0 conn(0x7f80a806bd30 legacy=0x7f80a80b5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.598+0000 7f80ace4f640 1 -- 172.21.15.62:0/2463102014 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a80b0210 con 0x7f80a80b64d0 2024-04-18T10:06:24.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f80ace4f640 1 --2- 172.21.15.62:0/2463102014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a80b64d0 0x7f80a814b780 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8098009870 tx=0x7f809801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856a96f81c12e015 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:24.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f809ffff640 1 -- 172.21.15.62:0/2463102014 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80980b0070 con 0x7f80a80b64d0 2024-04-18T10:06:24.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f809ffff640 1 -- 172.21.15.62:0/2463102014 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80980ab600 con 0x7f80a80b64d0 2024-04-18T10:06:24.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f809ffff640 1 -- 172.21.15.62:0/2463102014 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80980ab9d0 con 0x7f80a80b64d0 2024-04-18T10:06:24.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f80af8b4640 1 -- 172.21.15.62:0/2463102014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a80b64d0 msgr2=0x7f80a814b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f80af8b4640 1 --2- 172.21.15.62:0/2463102014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a80b64d0 0x7f80a814b780 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8098009870 tx=0x7f809801c4d0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f80af8b4640 1 -- 172.21.15.62:0/2463102014 shutdown_connections 2024-04-18T10:06:24.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f80af8b4640 1 --2- 172.21.15.62:0/2463102014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a806c110 0x7f80a814bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f80af8b4640 1 --2- 172.21.15.62:0/2463102014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a80b64d0 0x7f80a814b780 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f80af8b4640 1 -- 172.21.15.62:0/2463102014 >> 172.21.15.62:0/2463102014 conn(0x7f80a80ba710 msgr2=0x7f80a80bab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:24.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f80af8b4640 1 -- 172.21.15.62:0/2463102014 shutdown_connections 2024-04-18T10:06:24.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f80af8b4640 1 -- 172.21.15.62:0/2463102014 wait complete. 2024-04-18T10:06:24.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f80af8b4640 1 Processor -- start 2024-04-18T10:06:24.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.602+0000 7f80af8b4640 1 -- start start 2024-04-18T10:06:24.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80af8b4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a806bd30 0x7f80a814b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80af8b4640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80a806c110 0x7f80a81e4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80af8b4640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a80b64d0 0x7f80a80b4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80af8b4640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f80a8155470 con 0x7f80a806bd30 2024-04-18T10:06:24.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80af8b4640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f80a814c610 con 0x7f80a806c110 2024-04-18T10:06:24.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80af8b4640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f80a81555f0 con 0x7f80a80b64d0 2024-04-18T10:06:24.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ad650640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a806bd30 0x7f80a814b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ad650640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a806bd30 0x7f80a814b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37698/0 (socket says 172.21.15.62:37698) 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ad650640 1 -- 172.21.15.62:0/3480833230 learned_addr learned my addr 172.21.15.62:0/3480833230 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ade51640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a80b64d0 0x7f80a80b4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ace4f640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80a806c110 0x7f80a81e4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ad650640 1 -- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a80b64d0 msgr2=0x7f80a80b4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ad650640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a80b64d0 0x7f80a80b4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ad650640 1 -- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80a806c110 msgr2=0x7f80a81e4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ad650640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80a806c110 0x7f80a81e4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ad650640 1 -- 172.21.15.62:0/3480833230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a81e4580 con 0x7f80a806bd30 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ace4f640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80a806c110 0x7f80a81e4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ade51640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a80b64d0 0x7f80a80b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ad650640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a806bd30 0x7f80a814b090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f80a400c730 tx=0x7f80a400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:24.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f809dffb640 1 -- 172.21.15.62:0/3480833230 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a4016020 con 0x7f80a806bd30 2024-04-18T10:06:24.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f809dffb640 1 -- 172.21.15.62:0/3480833230 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80a4006d60 con 0x7f80a806bd30 2024-04-18T10:06:24.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80af8b4640 1 -- 172.21.15.62:0/3480833230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80a814da60 con 0x7f80a806bd30 2024-04-18T10:06:24.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f809dffb640 1 -- 172.21.15.62:0/3480833230 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a4003bc0 con 0x7f80a806bd30 2024-04-18T10:06:24.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80af8b4640 1 -- 172.21.15.62:0/3480833230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80a814ca70 con 0x7f80a806bd30 2024-04-18T10:06:24.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80af8b4640 1 -- 172.21.15.62:0/3480833230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80a8069230 con 0x7f80a806bd30 2024-04-18T10:06:24.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f809dffb640 1 -- 172.21.15.62:0/3480833230 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f80a400a850 con 0x7f80a806bd30 2024-04-18T10:06:24.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f809dffb640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f808c068380 0x7f808c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f809dffb640 1 -- 172.21.15.62:0/3480833230 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f80a4087340 con 0x7f80a806bd30 2024-04-18T10:06:24.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.606+0000 7f80ace4f640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f808c068380 0x7f808c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.610+0000 7f80ace4f640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f808c068380 0x7f808c06a840 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f80980ad000 tx=0x7f80980a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:24.610 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.610+0000 7f809dffb640 1 -- 172.21.15.62:0/3480833230 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f80a4020020 con 0x7f80a806bd30 2024-04-18T10:06:24.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 -- 172.21.15.62:0/3480833230 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f80a809b1b0 con 0x7f80a806bd30 2024-04-18T10:06:24.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f809dffb640 1 -- 172.21.15.62:0/3480833230 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f80a4023e40 con 0x7f80a806bd30 2024-04-18T10:06:24.772 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-04-18T10:06:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 -- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f808c068380 msgr2=0x7f808c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f808c068380 0x7f808c06a840 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f80980ad000 tx=0x7f80980a6000 comp rx=0 tx=0).stop 2024-04-18T10:06:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 -- 172.21.15.62:0/3480833230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a806bd30 msgr2=0x7f80a814b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a806bd30 0x7f80a814b090 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f80a400c730 tx=0x7f80a400cbf0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 -- 172.21.15.62:0/3480833230 shutdown_connections 2024-04-18T10:06:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f808c068380 0x7f808c06a840 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80a80b64d0 0x7f80a80b4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80a806c110 0x7f80a81e4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 --2- 172.21.15.62:0/3480833230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80a806bd30 0x7f80a814b090 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 -- 172.21.15.62:0/3480833230 >> 172.21.15.62:0/3480833230 conn(0x7f80a80ba710 msgr2=0x7f80a806eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:24.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 -- 172.21.15.62:0/3480833230 shutdown_connections 2024-04-18T10:06:24.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.774+0000 7f80af8b4640 1 -- 172.21.15.62:0/3480833230 wait complete. 2024-04-18T10:06:24.792 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-18T10:06:24.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764920c640 1 Processor -- start 2024-04-18T10:06:24.915 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764920c640 1 -- start start 2024-04-18T10:06:24.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764920c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76440b19c0 0x7f764413f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764920c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f764413fc00 0x7f7644150390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764920c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76440b3bc0 con 0x7f76440738b0 2024-04-18T10:06:24.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764920c640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f76440b2cd0 con 0x7f764413fc00 2024-04-18T10:06:24.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764920c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7644073c90 con 0x7f76440b19c0 2024-04-18T10:06:24.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764359e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f764413fc00 0x7f7644150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f7642d9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f76440738b0 0x7f76440b34b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35318/0 (socket says 172.21.15.62:35318) 2024-04-18T10:06:24.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f7642d9d640 1 -- 172.21.15.62:0/3420638693 learned_addr learned my addr 172.21.15.62:0/3420638693 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:24.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764259c640 1 --2- 172.21.15.62:0/3420638693 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76440b19c0 0x7f764413f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764359e640 1 -- 172.21.15.62:0/3420638693 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76440b19c0 msgr2=0x7f764413f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:24.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764359e640 1 --2- 172.21.15.62:0/3420638693 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76440b19c0 0x7f764413f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764359e640 1 -- 172.21.15.62:0/3420638693 >> v1:172.21.15.62:6789/0 conn(0x7f76440738b0 legacy=0x7f76440b34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f7641d9b640 1 -- 172.21.15.62:0/3420638693 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1997931836 0 0) 0x7f76440b3bc0 con 0x7f76440738b0 2024-04-18T10:06:24.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764359e640 1 -- 172.21.15.62:0/3420638693 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76440b4940 con 0x7f764413fc00 2024-04-18T10:06:24.918 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764259c640 1 --2- 172.21.15.62:0/3420638693 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76440b19c0 0x7f764413f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:24.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f764359e640 1 --2- 172.21.15.62:0/3420638693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f764413fc00 0x7f7644150390 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f76340097c0 tx=0x7f763401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fcc6bd86c48b101 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:24.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.918+0000 7f7641d9b640 1 -- 172.21.15.62:0/3420638693 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76340b2070 con 0x7f764413fc00 2024-04-18T10:06:24.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f7641d9b640 1 -- 172.21.15.62:0/3420638693 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76340ad640 con 0x7f764413fc00 2024-04-18T10:06:24.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f7641d9b640 1 -- 172.21.15.62:0/3420638693 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76340ada10 con 0x7f764413fc00 2024-04-18T10:06:24.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 -- 172.21.15.62:0/3420638693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f764413fc00 msgr2=0x7f7644150390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 --2- 172.21.15.62:0/3420638693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f764413fc00 0x7f7644150390 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f76340097c0 tx=0x7f763401c4a0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 -- 172.21.15.62:0/3420638693 shutdown_connections 2024-04-18T10:06:24.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 --2- 172.21.15.62:0/3420638693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f764413fc00 0x7f7644150390 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 --2- 172.21.15.62:0/3420638693 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76440b19c0 0x7f764413f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 -- 172.21.15.62:0/3420638693 >> 172.21.15.62:0/3420638693 conn(0x7f76440ba4c0 msgr2=0x7f76440ba8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:24.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 -- 172.21.15.62:0/3420638693 shutdown_connections 2024-04-18T10:06:24.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 -- 172.21.15.62:0/3420638693 wait complete. 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 Processor -- start 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 -- start start 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76440738b0 0x7f76441376b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76440b19c0 0x7f7644137bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f764413fc00 0x7f76441e17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f76440b3fa0 con 0x7f76440b19c0 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7644149000 con 0x7f764413fc00 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f76440b4120 con 0x7f76440738b0 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76440b19c0 0x7f7644137bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76440b19c0 0x7f7644137bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37716/0 (socket says 172.21.15.62:37716) 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764259c640 1 -- 172.21.15.62:0/3012882023 learned_addr learned my addr 172.21.15.62:0/3012882023 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764359e640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f764413fc00 0x7f76441e17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f7642d9d640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76440738b0 0x7f76441376b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764259c640 1 -- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76440738b0 msgr2=0x7f76441376b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764259c640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76440738b0 0x7f76441376b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764259c640 1 -- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f764413fc00 msgr2=0x7f76441e17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:24.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764259c640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f764413fc00 0x7f76441e17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:24.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764259c640 1 -- 172.21.15.62:0/3012882023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7644143880 con 0x7f76440b19c0 2024-04-18T10:06:24.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f7642d9d640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76440738b0 0x7f76441376b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:24.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764259c640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76440b19c0 0x7f7644137bf0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f762c00c7c0 tx=0x7f762c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:24.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f763b7fe640 1 -- 172.21.15.62:0/3012882023 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c007a30 con 0x7f76440b19c0 2024-04-18T10:06:24.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f763b7fe640 1 -- 172.21.15.62:0/3012882023 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f762c007bd0 con 0x7f76440b19c0 2024-04-18T10:06:24.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 -- 172.21.15.62:0/3012882023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7644145030 con 0x7f76440b19c0 2024-04-18T10:06:24.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f763b7fe640 1 -- 172.21.15.62:0/3012882023 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762c0113f0 con 0x7f76440b19c0 2024-04-18T10:06:24.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.922+0000 7f764920c640 1 -- 172.21.15.62:0/3012882023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7644140300 con 0x7f76440b19c0 2024-04-18T10:06:24.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.926+0000 7f764920c640 1 -- 172.21.15.62:0/3012882023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7644069230 con 0x7f76440b19c0 2024-04-18T10:06:24.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.926+0000 7f763b7fe640 1 -- 172.21.15.62:0/3012882023 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f762c011590 con 0x7f76440b19c0 2024-04-18T10:06:24.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.926+0000 7f763b7fe640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7624068380 0x7f762406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:24.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.926+0000 7f763b7fe640 1 -- 172.21.15.62:0/3012882023 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f762c086ba0 con 0x7f76440b19c0 2024-04-18T10:06:24.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.926+0000 7f7642d9d640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7624068380 0x7f762406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:24.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.926+0000 7f7642d9d640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7624068380 0x7f762406a840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f763c007fd0 tx=0x7f763c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:24.929 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:24.930+0000 7f763b7fe640 1 -- 172.21.15.62:0/3012882023 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f762c015020 con 0x7f76440b19c0 2024-04-18T10:06:25.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.102+0000 7f764920c640 1 -- 172.21.15.62:0/3012882023 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7644143330 con 0x7f76440b19c0 2024-04-18T10:06:25.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f763b7fe640 1 -- 172.21.15.62:0/3012882023 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f762c057150 con 0x7f76440b19c0 2024-04-18T10:06:25.103 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-04-18T10:06:25.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f764920c640 1 -- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7624068380 msgr2=0x7f762406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f764920c640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7624068380 0x7f762406a840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f763c007fd0 tx=0x7f763c009040 comp rx=0 tx=0).stop 2024-04-18T10:06:25.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f764920c640 1 -- 172.21.15.62:0/3012882023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76440b19c0 msgr2=0x7f7644137bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f764920c640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76440b19c0 0x7f7644137bf0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f762c00c7c0 tx=0x7f762c00cc80 comp rx=0 tx=0).stop 2024-04-18T10:06:25.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f764920c640 1 -- 172.21.15.62:0/3012882023 shutdown_connections 2024-04-18T10:06:25.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f764920c640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7624068380 0x7f762406a840 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f764920c640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f764413fc00 0x7f76441e17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f764920c640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76440b19c0 0x7f7644137bf0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f764920c640 1 --2- 172.21.15.62:0/3012882023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76440738b0 0x7f76441376b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f764920c640 1 -- 172.21.15.62:0/3012882023 >> 172.21.15.62:0/3012882023 conn(0x7f76440ba4c0 msgr2=0x7f76440b8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:25.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f764920c640 1 -- 172.21.15.62:0/3012882023 shutdown_connections 2024-04-18T10:06:25.106 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.106+0000 7f764920c640 1 -- 172.21.15.62:0/3012882023 wait complete. 2024-04-18T10:06:25.122 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-04-18T10:06:25.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:25.122 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:25.144 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:06:25.144 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T10:06:25.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:25.144 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-18T10:06:25.198 INFO:tasks.cephfs.cephfs_test_case:[] 2024-04-18T10:06:25.199 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:25.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.370+0000 7fc0c778c640 1 Processor -- start 2024-04-18T10:06:25.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.370+0000 7fc0c778c640 1 -- start start 2024-04-18T10:06:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c778c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c009c730 0x7fc0c009e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c778c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0c009cb10 con 0x7fc0c00afe00 2024-04-18T10:06:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c778c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0c00b02e0 con 0x7fc0c014b370 2024-04-18T10:06:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c778c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c00b9ea0 con 0x7fc0c009c730 2024-04-18T10:06:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c4d27640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc0c00afe00 0x7fc0c00986a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35326/0 (socket says 172.21.15.62:35326) 2024-04-18T10:06:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c4d27640 1 -- 172.21.15.62:0/3500156608 learned_addr learned my addr 172.21.15.62:0/3500156608 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c5528640 1 --2- 172.21.15.62:0/3500156608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c009c730 0x7fc0c009e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2212147140 0 0) 0x7fc0c009cb10 con 0x7fc0c00afe00 2024-04-18T10:06:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0ac003530 con 0x7fc0c00afe00 2024-04-18T10:06:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3217769903 0 0) 0x7fc0ac003530 con 0x7fc0c00afe00 2024-04-18T10:06:25.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c009c730 msgr2=0x7fc0c009e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 --2- 172.21.15.62:0/3500156608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c009c730 0x7fc0c009e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 >> v1:172.21.15.145:6789/0 conn(0x7fc0c014b370 legacy=0x7fc0c01526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c00b9b30 con 0x7fc0c00afe00 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fc0b0001680 con 0x7fc0c00afe00 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 >> v1:172.21.15.62:6789/0 conn(0x7fc0c00afe00 legacy=0x7fc0c00986a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 --2- 172.21.15.62:0/3500156608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c009c730 0x7fc0ac0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 --2- 172.21.15.62:0/3500156608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0ac003fc0 0x7fc0ac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 --2- 172.21.15.62:0/3500156608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0ac004d50 0x7fc0ac0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c00b9ea0 con 0x7fc0ac004d50 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fc0ac00bb10 con 0x7fc0ac003fc0 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc0ac00bd10 con 0x7fc0c009c730 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c5d29640 1 --2- 172.21.15.62:0/3500156608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0ac004d50 0x7fc0ac0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c4d27640 1 --2- 172.21.15.62:0/3500156608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0ac003fc0 0x7fc0ac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c5528640 1 --2- 172.21.15.62:0/3500156608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c009c730 0x7fc0ac0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c5d29640 1 -- 172.21.15.62:0/3500156608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c009c730 msgr2=0x7fc0ac0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c5d29640 1 --2- 172.21.15.62:0/3500156608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c009c730 0x7fc0ac0039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c5d29640 1 -- 172.21.15.62:0/3500156608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0ac003fc0 msgr2=0x7fc0ac004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c5d29640 1 --2- 172.21.15.62:0/3500156608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0ac003fc0 0x7fc0ac004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c5d29640 1 -- 172.21.15.62:0/3500156608 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0ac003530 con 0x7fc0ac004d50 2024-04-18T10:06:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0c5d29640 1 --2- 172.21.15.62:0/3500156608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0ac004d50 0x7fc0ac0051d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc001e70 tx=0x7fc0bc01c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea52d6e91fef1cd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc001a90 con 0x7fc0ac004d50 2024-04-18T10:06:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0bc003b80 con 0x7fc0ac004d50 2024-04-18T10:06:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.374+0000 7fc0b7fff640 1 -- 172.21.15.62:0/3500156608 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc005080 con 0x7fc0ac004d50 2024-04-18T10:06:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 -- 172.21.15.62:0/3500156608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0ac004d50 msgr2=0x7fc0ac0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 --2- 172.21.15.62:0/3500156608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0ac004d50 0x7fc0ac0051d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc001e70 tx=0x7fc0bc01c780 comp rx=0 tx=0).stop 2024-04-18T10:06:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c5d29640 1 -- 172.21.15.62:0/3500156608 reap_dead start 2024-04-18T10:06:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 -- 172.21.15.62:0/3500156608 shutdown_connections 2024-04-18T10:06:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 -- 172.21.15.62:0/3500156608 >> 172.21.15.62:0/3500156608 conn(0x7fc0c006c0d0 msgr2=0x7fc0c00b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 -- 172.21.15.62:0/3500156608 shutdown_connections 2024-04-18T10:06:25.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 -- 172.21.15.62:0/3500156608 wait complete. 2024-04-18T10:06:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 Processor -- start 2024-04-18T10:06:25.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 -- start start 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0c0072480 0x7fc0c0072860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c0073080 0x7fc0c01dd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0c01ddec0 0x7fc0c0145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c0098f30 con 0x7fc0c0072480 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c0098db0 con 0x7fc0c01ddec0 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c009e0f0 con 0x7fc0c0073080 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c5528640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0c0072480 0x7fc0c0072860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c5528640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0c0072480 0x7fc0c0072860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37734/0 (socket says 172.21.15.62:37734) 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c5528640 1 -- 172.21.15.62:0/497523347 learned_addr learned my addr 172.21.15.62:0/497523347 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c4d27640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c0073080 0x7fc0c01dd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c5528640 1 -- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c0073080 msgr2=0x7fc0c01dd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c5528640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c0073080 0x7fc0c01dd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c5d29640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0c01ddec0 0x7fc0c0145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c5528640 1 -- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0c01ddec0 msgr2=0x7fc0c0145080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c5528640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0c01ddec0 0x7fc0c0145080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c5528640 1 -- 172.21.15.62:0/497523347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c01de350 con 0x7fc0c0072480 2024-04-18T10:06:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c4d27640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c0073080 0x7fc0c01dd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c5528640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0c0072480 0x7fc0c0072860 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8010890 tx=0x7fc0b8010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0b5ffb640 1 -- 172.21.15.62:0/497523347 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b800f040 con 0x7fc0c0072480 2024-04-18T10:06:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0b5ffb640 1 -- 172.21.15.62:0/497523347 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b800eb60 con 0x7fc0c0072480 2024-04-18T10:06:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 -- 172.21.15.62:0/497523347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c00b0e40 con 0x7fc0c0072480 2024-04-18T10:06:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0b5ffb640 1 -- 172.21.15.62:0/497523347 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b801c400 con 0x7fc0c0072480 2024-04-18T10:06:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.378+0000 7fc0c778c640 1 -- 172.21.15.62:0/497523347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c0070fd0 con 0x7fc0c0072480 2024-04-18T10:06:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.382+0000 7fc0c778c640 1 -- 172.21.15.62:0/497523347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c0068b90 con 0x7fc0c0072480 2024-04-18T10:06:25.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.382+0000 7fc0b5ffb640 1 -- 172.21.15.62:0/497523347 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc0b801c5a0 con 0x7fc0c0072480 2024-04-18T10:06:25.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.382+0000 7fc0b5ffb640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc0a4068380 0x7fc0a406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.382+0000 7fc0b5ffb640 1 -- 172.21.15.62:0/497523347 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc0b8087000 con 0x7fc0c0072480 2024-04-18T10:06:25.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.382+0000 7fc0c4d27640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc0a4068380 0x7fc0a406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.382+0000 7fc0c4d27640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc0a4068380 0x7fc0a406a840 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc0c00988c0 tx=0x7fc0b0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:25.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.386+0000 7fc0b5ffb640 1 -- 172.21.15.62:0/497523347 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc0b8018030 con 0x7fc0c0072480 2024-04-18T10:06:25.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.574+0000 7fc0c778c640 1 -- 172.21.15.62:0/497523347 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0c00b4e30 con 0x7fc0c0072480 2024-04-18T10:06:25.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0b5ffb640 1 -- 172.21.15.62:0/497523347 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3527 (secure 0 0 0) 0x7fc0b8057650 con 0x7fc0c0072480 2024-04-18T10:06:25.575 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:25.576 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/904929673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":904929673},{"type":"v1","addr":"172.21.15.145:6835","nonce":904929673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:05:58.588100+0000","modified":"2024-04-18T10:06:01.605447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2120622858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2120622858},{"type":"v1","addr":"172.21.15.62:6833","nonce":2120622858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-18T10:06:25.576 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-04-18T10:06:25.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0c778c640 1 -- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc0a4068380 msgr2=0x7fc0a406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0c778c640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc0a4068380 0x7fc0a406a840 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc0c00988c0 tx=0x7fc0b0009040 comp rx=0 tx=0).stop 2024-04-18T10:06:25.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0c778c640 1 -- 172.21.15.62:0/497523347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0c0072480 msgr2=0x7fc0c0072860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0c778c640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0c0072480 0x7fc0c0072860 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8010890 tx=0x7fc0b8010d50 comp rx=0 tx=0).stop 2024-04-18T10:06:25.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0c778c640 1 -- 172.21.15.62:0/497523347 shutdown_connections 2024-04-18T10:06:25.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0c778c640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc0a4068380 0x7fc0a406a840 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0c778c640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0c01ddec0 0x7fc0c0145080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0c778c640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0c0073080 0x7fc0c01dd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0c778c640 1 --2- 172.21.15.62:0/497523347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0c0072480 0x7fc0c0072860 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0c778c640 1 -- 172.21.15.62:0/497523347 >> 172.21.15.62:0/497523347 conn(0x7fc0c006c0d0 msgr2=0x7fc0c00b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:25.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0c778c640 1 -- 172.21.15.62:0/497523347 shutdown_connections 2024-04-18T10:06:25.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.578+0000 7fc0c778c640 1 -- 172.21.15.62:0/497523347 wait complete. 2024-04-18T10:06:25.593 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:25.715 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9ce818640 1 Processor -- start 2024-04-18T10:06:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9ce818640 1 -- start start 2024-04-18T10:06:25.716 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9ce818640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c80b6360 0x7fb9c8148640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9ce818640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb9c80b7f70 0x7fb9c8141900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9ce818640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9c8148b80 con 0x7fb9c8141e40 2024-04-18T10:06:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9ce818640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fb9c80b6740 con 0x7fb9c80b7f70 2024-04-18T10:06:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9ce818640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fb9c80b7720 con 0x7fb9c80b6360 2024-04-18T10:06:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9ccdb5640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb9c8141e40 0x7fb9c814b220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35334/0 (socket says 172.21.15.62:35334) 2024-04-18T10:06:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9ccdb5640 1 -- 172.21.15.62:0/2226790828 learned_addr learned my addr 172.21.15.62:0/2226790828 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c7fff640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c80b6360 0x7fb9c8148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c77fe640 1 --2- 172.21.15.62:0/2226790828 >> v2:172.21.15.145:3300/0 conn(0x7fb9c80b7f70 0x7fb9c8141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1613025879 0 0) 0x7fb9c8148b80 con 0x7fb9c8141e40 2024-04-18T10:06:25.717 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9b4003530 con 0x7fb9c8141e40 2024-04-18T10:06:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2832216553 0 0) 0x7fb9b4003530 con 0x7fb9c8141e40 2024-04-18T10:06:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c80b6360 msgr2=0x7fb9c8148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c80b6360 0x7fb9c8148640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 >> v2:172.21.15.145:3300/0 conn(0x7fb9c80b7f70 msgr2=0x7fb9c8141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 --2- 172.21.15.62:0/2226790828 >> v2:172.21.15.145:3300/0 conn(0x7fb9c80b7f70 0x7fb9c8141900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c8147e40 con 0x7fb9c8141e40 2024-04-18T10:06:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fb9c0001680 con 0x7fb9c8141e40 2024-04-18T10:06:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 >> v1:172.21.15.62:6789/0 conn(0x7fb9c8141e40 legacy=0x7fb9c814b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.719 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b40039c0 0x7fb9b4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9b4004810 0x7fb9b4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c80b6360 0x7fb9b40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c80b6740 con 0x7fb9b40039c0 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c80b7720 con 0x7fb9b4004810 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fb9b400b9d0 con 0x7fb9c80b6360 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c7fff640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b40039c0 0x7fb9b4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c77fe640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9b4004810 0x7fb9b4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9ccdb5640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c80b6360 0x7fb9b40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c7fff640 1 -- 172.21.15.62:0/2226790828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c80b6360 msgr2=0x7fb9b40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c7fff640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c80b6360 0x7fb9b40051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c7fff640 1 -- 172.21.15.62:0/2226790828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9b4004810 msgr2=0x7fb9b4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c7fff640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9b4004810 0x7fb9b4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c7fff640 1 -- 172.21.15.62:0/2226790828 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b4003530 con 0x7fb9b40039c0 2024-04-18T10:06:25.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.718+0000 7fb9c77fe640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9b4004810 0x7fb9b4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c7fff640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b40039c0 0x7fb9b4003de0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009b10 tx=0x7fb9b001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00967c5b2d4d159 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b00096b0 con 0x7fb9b40039c0 2024-04-18T10:06:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9b00a9470 con 0x7fb9b40039c0 2024-04-18T10:06:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c6ffd640 1 -- 172.21.15.62:0/2226790828 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b00a9860 con 0x7fb9b40039c0 2024-04-18T10:06:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 -- 172.21.15.62:0/2226790828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b40039c0 msgr2=0x7fb9b4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 --2- 172.21.15.62:0/2226790828 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b40039c0 0x7fb9b4003de0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb9b0009b10 tx=0x7fb9b001ca70 comp rx=0 tx=0).stop 2024-04-18T10:06:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ccdb5640 1 -- 172.21.15.62:0/2226790828 reap_dead start 2024-04-18T10:06:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 -- 172.21.15.62:0/2226790828 shutdown_connections 2024-04-18T10:06:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 -- 172.21.15.62:0/2226790828 >> 172.21.15.62:0/2226790828 conn(0x7fb9c80ba070 msgr2=0x7fb9c80ba480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 -- 172.21.15.62:0/2226790828 shutdown_connections 2024-04-18T10:06:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 -- 172.21.15.62:0/2226790828 wait complete. 2024-04-18T10:06:25.722 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 Processor -- start 2024-04-18T10:06:25.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 -- start start 2024-04-18T10:06:25.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b40039c0 0x7fb9c8146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9c8142ac0 0x7fb9c8144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c81e3c40 0x7fb9c81e40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c80b50c0 con 0x7fb9b40039c0 2024-04-18T10:06:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fb9c80b4f40 con 0x7fb9c8142ac0 2024-04-18T10:06:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ce818640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fb9c80b7240 con 0x7fb9c81e3c40 2024-04-18T10:06:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b40039c0 0x7fb9c8146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b40039c0 0x7fb9c8146e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37754/0 (socket says 172.21.15.62:37754) 2024-04-18T10:06:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c7fff640 1 -- 172.21.15.62:0/3289329604 learned_addr learned my addr 172.21.15.62:0/3289329604 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9ccdb5640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c81e3c40 0x7fb9c81e40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c77fe640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9c8142ac0 0x7fb9c8144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c7fff640 1 -- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c81e3c40 msgr2=0x7fb9c81e40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c7fff640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c81e3c40 0x7fb9c81e40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c7fff640 1 -- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9c8142ac0 msgr2=0x7fb9c8144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c7fff640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9c8142ac0 0x7fb9c8144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c7fff640 1 -- 172.21.15.62:0/3289329604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9c8143510 con 0x7fb9b40039c0 2024-04-18T10:06:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.722+0000 7fb9c77fe640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9c8142ac0 0x7fb9c8144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.726+0000 7fb9c7fff640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b40039c0 0x7fb9c8146e50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb9b00ab040 tx=0x7fb9b0019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.726+0000 7fb9c4ff9640 1 -- 172.21.15.62:0/3289329604 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b00bf020 con 0x7fb9b40039c0 2024-04-18T10:06:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.726+0000 7fb9c4ff9640 1 -- 172.21.15.62:0/3289329604 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9b00a9c40 con 0x7fb9b40039c0 2024-04-18T10:06:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.726+0000 7fb9ce818640 1 -- 172.21.15.62:0/3289329604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9c8142f50 con 0x7fb9b40039c0 2024-04-18T10:06:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.726+0000 7fb9c4ff9640 1 -- 172.21.15.62:0/3289329604 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9b0004020 con 0x7fb9b40039c0 2024-04-18T10:06:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.726+0000 7fb9ce818640 1 -- 172.21.15.62:0/3289329604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9c8144280 con 0x7fb9b40039c0 2024-04-18T10:06:25.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.726+0000 7fb9ce818640 1 -- 172.21.15.62:0/3289329604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9c8069230 con 0x7fb9b40039c0 2024-04-18T10:06:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.726+0000 7fb9c4ff9640 1 -- 172.21.15.62:0/3289329604 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb9b00c2070 con 0x7fb9b40039c0 2024-04-18T10:06:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.726+0000 7fb9c4ff9640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb9a8068380 0x7fb9a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.726+0000 7fb9c4ff9640 1 -- 172.21.15.62:0/3289329604 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb9b0120de0 con 0x7fb9b40039c0 2024-04-18T10:06:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.726+0000 7fb9c77fe640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb9a8068380 0x7fb9a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:25.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.730+0000 7fb9c77fe640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb9a8068380 0x7fb9a806a840 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb9c81441b0 tx=0x7fb9b800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:25.729 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.730+0000 7fb9c4ff9640 1 -- 172.21.15.62:0/3289329604 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb9b00c8020 con 0x7fb9b40039c0 2024-04-18T10:06:25.893 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.894+0000 7fb9ce818640 1 -- 172.21.15.62:0/3289329604 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb9c8145650 con 0x7fb9b40039c0 2024-04-18T10:06:25.895 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.894+0000 7fb9c4ff9640 1 -- 172.21.15.62:0/3289329604 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7fb9b00ad020 con 0x7fb9b40039c0 2024-04-18T10:06:25.895 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:25.895 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:25.897 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.898+0000 7fb9ce818640 1 -- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb9a8068380 msgr2=0x7fb9a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.898+0000 7fb9ce818640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb9a8068380 0x7fb9a806a840 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb9c81441b0 tx=0x7fb9b800b040 comp rx=0 tx=0).stop 2024-04-18T10:06:25.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.898+0000 7fb9ce818640 1 -- 172.21.15.62:0/3289329604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b40039c0 msgr2=0x7fb9c8146e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:25.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.898+0000 7fb9ce818640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b40039c0 0x7fb9c8146e50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb9b00ab040 tx=0x7fb9b0019810 comp rx=0 tx=0).stop 2024-04-18T10:06:25.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.898+0000 7fb9ce818640 1 -- 172.21.15.62:0/3289329604 shutdown_connections 2024-04-18T10:06:25.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.898+0000 7fb9ce818640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb9a8068380 0x7fb9a806a840 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.898+0000 7fb9ce818640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9c81e3c40 0x7fb9c81e40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.898+0000 7fb9ce818640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9c8142ac0 0x7fb9c8144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.898+0000 7fb9ce818640 1 --2- 172.21.15.62:0/3289329604 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9b40039c0 0x7fb9c8146e50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:25.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.898+0000 7fb9ce818640 1 -- 172.21.15.62:0/3289329604 >> 172.21.15.62:0/3289329604 conn(0x7fb9c80ba070 msgr2=0x7fb9c8140950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:25.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.898+0000 7fb9ce818640 1 -- 172.21.15.62:0/3289329604 shutdown_connections 2024-04-18T10:06:25.898 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:25.898+0000 7fb9ce818640 1 -- 172.21.15.62:0/3289329604 wait complete. 2024-04-18T10:06:25.915 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-18T10:05:56.782649+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-18T10:05:57.541623+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:25.915 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:26.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbc0263640 1 Processor -- start 2024-04-18T10:06:26.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbc0263640 1 -- start start 2024-04-18T10:06:26.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbc0263640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcbb80b01a0 0x7fcbb806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbc0263640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcbb80ae4e0 0x7fcbb813e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbc0263640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbb80737c0 0x7fcbb813eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbc0263640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fcbb80b0580 con 0x7fcbb80b01a0 2024-04-18T10:06:26.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbc0263640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fcbb80b59d0 con 0x7fcbb80ae4e0 2024-04-18T10:06:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbc0263640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fcbb80ae970 con 0x7fcbb80737c0 2024-04-18T10:06:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbbdfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcbb80b01a0 0x7fcbb806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbbdfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcbb80b01a0 0x7fcbb806d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37764/0 (socket says 172.21.15.62:37764) 2024-04-18T10:06:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbbdfff640 1 -- 172.21.15.62:0/2574494514 learned_addr learned my addr 172.21.15.62:0/2574494514 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbbe800640 1 --2- 172.21.15.62:0/2574494514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbb80737c0 0x7fcbb813eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbbd7fe640 1 --2- 172.21.15.62:0/2574494514 >> v2:172.21.15.145:3300/0 conn(0x7fcbb80ae4e0 0x7fcbb813e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbbdfff640 1 -- 172.21.15.62:0/2574494514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbb80737c0 msgr2=0x7fcbb813eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbbdfff640 1 --2- 172.21.15.62:0/2574494514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbb80737c0 0x7fcbb813eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbbdfff640 1 -- 172.21.15.62:0/2574494514 >> v2:172.21.15.145:3300/0 conn(0x7fcbb80ae4e0 msgr2=0x7fcbb813e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbbdfff640 1 --2- 172.21.15.62:0/2574494514 >> v2:172.21.15.145:3300/0 conn(0x7fcbb80ae4e0 0x7fcbb813e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.046+0000 7fcbbdfff640 1 -- 172.21.15.62:0/2574494514 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb80b53a0 con 0x7fcbb80b01a0 2024-04-18T10:06:26.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbbdfff640 1 --2- 172.21.15.62:0/2574494514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb80b01a0 0x7fcbb806d2a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcbb806ec10 tx=0x7fcbb401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b369c560fc967643 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:26.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbbcffd640 1 -- 172.21.15.62:0/2574494514 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb40b0070 con 0x7fcbb80b01a0 2024-04-18T10:06:26.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbbcffd640 1 -- 172.21.15.62:0/2574494514 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb40ab600 con 0x7fcbb80b01a0 2024-04-18T10:06:26.049 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbbcffd640 1 -- 172.21.15.62:0/2574494514 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb40ab9d0 con 0x7fcbb80b01a0 2024-04-18T10:06:26.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbc0263640 1 -- 172.21.15.62:0/2574494514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb80b01a0 msgr2=0x7fcbb806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbc0263640 1 --2- 172.21.15.62:0/2574494514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb80b01a0 0x7fcbb806d2a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcbb806ec10 tx=0x7fcbb401c400 comp rx=0 tx=0).stop 2024-04-18T10:06:26.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbc0263640 1 -- 172.21.15.62:0/2574494514 shutdown_connections 2024-04-18T10:06:26.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbc0263640 1 --2- 172.21.15.62:0/2574494514 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbb80737c0 0x7fcbb813eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbc0263640 1 --2- 172.21.15.62:0/2574494514 >> v2:172.21.15.145:3300/0 conn(0x7fcbb80ae4e0 0x7fcbb813e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbc0263640 1 --2- 172.21.15.62:0/2574494514 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb80b01a0 0x7fcbb806d2a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbc0263640 1 -- 172.21.15.62:0/2574494514 >> 172.21.15.62:0/2574494514 conn(0x7fcbb80b9690 msgr2=0x7fcbb80b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:26.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbc0263640 1 -- 172.21.15.62:0/2574494514 shutdown_connections 2024-04-18T10:06:26.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.050+0000 7fcbc0263640 1 -- 172.21.15.62:0/2574494514 wait complete. 2024-04-18T10:06:26.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbc0263640 1 Processor -- start 2024-04-18T10:06:26.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbc0263640 1 -- start start 2024-04-18T10:06:26.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbc0263640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbb80737c0 0x7fcbb809d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbc0263640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbb80b01a0 0x7fcbb81433f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbc0263640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb8145310 0x7fcbb81e0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbc0263640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb8148910 con 0x7fcbb8145310 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbc0263640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb806e470 con 0x7fcbb80737c0 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbc0263640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fcbb8148a90 con 0x7fcbb80b01a0 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbbe800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb8145310 0x7fcbb81e0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbbe800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb8145310 0x7fcbb81e0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37778/0 (socket says 172.21.15.62:37778) 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbbe800640 1 -- 172.21.15.62:0/3641000342 learned_addr learned my addr 172.21.15.62:0/3641000342 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbbd7fe640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbb80b01a0 0x7fcbb81433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbbdfff640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbb80737c0 0x7fcbb809d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbbe800640 1 -- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbb80b01a0 msgr2=0x7fcbb81433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbbe800640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbb80b01a0 0x7fcbb81433f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbbe800640 1 -- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbb80737c0 msgr2=0x7fcbb809d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbbe800640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbb80737c0 0x7fcbb809d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbbe800640 1 -- 172.21.15.62:0/3641000342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb8144a10 con 0x7fcbb8145310 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbbd7fe640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbb80b01a0 0x7fcbb81433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:26.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbbe800640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb8145310 0x7fcbb81e0e90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0007e60 tx=0x7fcbb000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:26.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbae7fc640 1 -- 172.21.15.62:0/3641000342 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0014620 con 0x7fcbb8145310 2024-04-18T10:06:26.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbae7fc640 1 -- 172.21.15.62:0/3641000342 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbb0019070 con 0x7fcbb8145310 2024-04-18T10:06:26.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbc0263640 1 -- 172.21.15.62:0/3641000342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb809de10 con 0x7fcbb8145310 2024-04-18T10:06:26.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbae7fc640 1 -- 172.21.15.62:0/3641000342 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0014a20 con 0x7fcbb8145310 2024-04-18T10:06:26.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbc0263640 1 -- 172.21.15.62:0/3641000342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb8144730 con 0x7fcbb8145310 2024-04-18T10:06:26.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbc0263640 1 -- 172.21.15.62:0/3641000342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb809e330 con 0x7fcbb8145310 2024-04-18T10:06:26.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.054+0000 7fcbae7fc640 1 -- 172.21.15.62:0/3641000342 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcbb0014bc0 con 0x7fcbb8145310 2024-04-18T10:06:26.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.058+0000 7fcbae7fc640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcb9c068380 0x7fcb9c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.058+0000 7fcbae7fc640 1 -- 172.21.15.62:0/3641000342 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fcbb00870d0 con 0x7fcbb8145310 2024-04-18T10:06:26.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.058+0000 7fcbbdfff640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcb9c068380 0x7fcb9c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.056 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.058+0000 7fcbbdfff640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcb9c068380 0x7fcb9c06a840 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40ad000 tx=0x7fcbb40a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:26.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.062+0000 7fcbae7fc640 1 -- 172.21.15.62:0/3641000342 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcbb809e330 con 0x7fcbb8145310 2024-04-18T10:06:26.223 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.222+0000 7fcbc0263640 1 -- 172.21.15.62:0/3641000342 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcbb8068c00 con 0x7fcbb8145310 2024-04-18T10:06:26.224 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbae7fc640 1 -- 172.21.15.62:0/3641000342 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7fcbb0057720 con 0x7fcbb8145310 2024-04-18T10:06:26.224 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbc0263640 1 -- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcb9c068380 msgr2=0x7fcb9c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbc0263640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcb9c068380 0x7fcb9c06a840 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcbb40ad000 tx=0x7fcbb40a6000 comp rx=0 tx=0).stop 2024-04-18T10:06:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbc0263640 1 -- 172.21.15.62:0/3641000342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb8145310 msgr2=0x7fcbb81e0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbc0263640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb8145310 0x7fcbb81e0e90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0007e60 tx=0x7fcbb000e4b0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbc0263640 1 -- 172.21.15.62:0/3641000342 shutdown_connections 2024-04-18T10:06:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbc0263640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcb9c068380 0x7fcb9c06a840 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbc0263640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbb8145310 0x7fcbb81e0e90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbc0263640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbb80b01a0 0x7fcbb81433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbc0263640 1 --2- 172.21.15.62:0/3641000342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbb80737c0 0x7fcbb809d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbc0263640 1 -- 172.21.15.62:0/3641000342 >> 172.21.15.62:0/3641000342 conn(0x7fcbb80b9690 msgr2=0x7fcbb8136ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:26.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbc0263640 1 -- 172.21.15.62:0/3641000342 shutdown_connections 2024-04-18T10:06:26.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.226+0000 7fcbc0263640 1 -- 172.21.15.62:0/3641000342 wait complete. 2024-04-18T10:06:26.243 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-18T10:06:26.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8d01da640 1 Processor -- start 2024-04-18T10:06:26.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8d01da640 1 -- start start 2024-04-18T10:06:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8d01da640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc8c809bd40 0x7fc8c80b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8d01da640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8c806f140 con 0x7fc8c80b70b0 2024-04-18T10:06:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8d01da640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8c80b6870 con 0x7fc8c80b7530 2024-04-18T10:06:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8d01da640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fc8c80af5d0 con 0x7fc8c809bd40 2024-04-18T10:06:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8cd775640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc8c80b70b0 0x7fc8c806e320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35344/0 (socket says 172.21.15.62:35344) 2024-04-18T10:06:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8cd775640 1 -- 172.21.15.62:0/3697540933 learned_addr learned my addr 172.21.15.62:0/3697540933 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8cdf76640 1 --2- 172.21.15.62:0/3697540933 >> v2:172.21.15.145:3300/0 conn(0x7fc8c809bd40 0x7fc8c80b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 856408191 0 0) 0x7fc8c806f140 con 0x7fc8c80b70b0 2024-04-18T10:06:26.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8b4003530 con 0x7fc8c80b70b0 2024-04-18T10:06:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2663747871 0 0) 0x7fc8b4003530 con 0x7fc8c80b70b0 2024-04-18T10:06:26.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 >> v2:172.21.15.145:3300/0 conn(0x7fc8c809bd40 msgr2=0x7fc8c80b4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 --2- 172.21.15.62:0/3697540933 >> v2:172.21.15.145:3300/0 conn(0x7fc8c809bd40 0x7fc8c80b4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 >> v1:172.21.15.145:6789/0 conn(0x7fc8c80b7530 legacy=0x7fc8c806ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c809c120 con 0x7fc8c80b70b0 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fc8b8001680 con 0x7fc8c80b70b0 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 >> v1:172.21.15.62:6789/0 conn(0x7fc8c80b70b0 legacy=0x7fc8c806e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 0x7fc8b4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8b4004810 0x7fc8b4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8b40052f0 0x7fc8b400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b400c190 con 0x7fc8b40039c0 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b400c340 con 0x7fc8b40052f0 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc8b400c540 con 0x7fc8b4004810 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8cdf76640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 0x7fc8b4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8cd775640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8b4004810 0x7fc8b4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ce777640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8b40052f0 0x7fc8b400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8cdf76640 1 -- 172.21.15.62:0/3697540933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8b4004810 msgr2=0x7fc8b4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8cdf76640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8b4004810 0x7fc8b4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8cdf76640 1 -- 172.21.15.62:0/3697540933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8b40052f0 msgr2=0x7fc8b400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8cdf76640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8b40052f0 0x7fc8b400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8cdf76640 1 -- 172.21.15.62:0/3697540933 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c809c120 con 0x7fc8b40039c0 2024-04-18T10:06:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8cd775640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8b4004810 0x7fc8b4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ce777640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8b40052f0 0x7fc8b400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:26.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.374+0000 7fc8ce777640 1 -- 172.21.15.62:0/3697540933 reap_dead start 2024-04-18T10:06:26.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8cdf76640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 0x7fc8b4003e00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009ae0 tx=0x7fc8c401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459c2a6d08db5882 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:26.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c40096b0 con 0x7fc8b40039c0 2024-04-18T10:06:26.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c40a6450 con 0x7fc8b40039c0 2024-04-18T10:06:26.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8ccf74640 1 -- 172.21.15.62:0/3697540933 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c40a6840 con 0x7fc8b40039c0 2024-04-18T10:06:26.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 -- 172.21.15.62:0/3697540933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 msgr2=0x7fc8b4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 0x7fc8b4003e00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009ae0 tx=0x7fc8c401ca70 comp rx=0 tx=0).stop 2024-04-18T10:06:26.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 -- 172.21.15.62:0/3697540933 shutdown_connections 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 --2- 172.21.15.62:0/3697540933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 0x7fc8b4003e00 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 -- 172.21.15.62:0/3697540933 >> 172.21.15.62:0/3697540933 conn(0x7fc8c80ba780 msgr2=0x7fc8c80bab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 -- 172.21.15.62:0/3697540933 shutdown_connections 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 -- 172.21.15.62:0/3697540933 wait complete. 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 Processor -- start 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 -- start start 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 0x7fc8c8122190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8c809ecc0 0x7fc8c80b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8c81e1740 0x7fc8c8146310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c806eb30 con 0x7fc8b40039c0 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fc8c80b6ba0 con 0x7fc8c809ecc0 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8d01da640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc8c806ecb0 con 0x7fc8c81e1740 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8cdf76640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 0x7fc8c8122190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8cdf76640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 0x7fc8c8122190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37810/0 (socket says 172.21.15.62:37810) 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.378+0000 7fc8cdf76640 1 -- 172.21.15.62:0/3692708758 learned_addr learned my addr 172.21.15.62:0/3692708758 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8cd775640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8c809ecc0 0x7fc8c80b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8ce777640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8c81e1740 0x7fc8c8146310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8cdf76640 1 -- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8c81e1740 msgr2=0x7fc8c8146310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8cdf76640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8c81e1740 0x7fc8c8146310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8cdf76640 1 -- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8c809ecc0 msgr2=0x7fc8c80b2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8cdf76640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8c809ecc0 0x7fc8c80b2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8cdf76640 1 -- 172.21.15.62:0/3692708758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8c809d830 con 0x7fc8b40039c0 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8cd775640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8c809ecc0 0x7fc8c80b2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8cdf76640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 0x7fc8c8122190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009ae0 tx=0x7fc8c4019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8be7fc640 1 -- 172.21.15.62:0/3692708758 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c40a6c80 con 0x7fc8b40039c0 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8be7fc640 1 -- 172.21.15.62:0/3692708758 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8c40a6e20 con 0x7fc8b40039c0 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8be7fc640 1 -- 172.21.15.62:0/3692708758 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8c40b9660 con 0x7fc8b40039c0 2024-04-18T10:06:26.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8d01da640 1 -- 172.21.15.62:0/3692708758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8c809d550 con 0x7fc8b40039c0 2024-04-18T10:06:26.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8d01da640 1 -- 172.21.15.62:0/3692708758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8c809e490 con 0x7fc8b40039c0 2024-04-18T10:06:26.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8d01da640 1 -- 172.21.15.62:0/3692708758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8c8069230 con 0x7fc8b40039c0 2024-04-18T10:06:26.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8be7fc640 1 -- 172.21.15.62:0/3692708758 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc8c40c2440 con 0x7fc8b40039c0 2024-04-18T10:06:26.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8be7fc640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc8ac0683b0 0x7fc8ac06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8be7fc640 1 -- 172.21.15.62:0/3692708758 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc8c4120ba0 con 0x7fc8b40039c0 2024-04-18T10:06:26.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.382+0000 7fc8cd775640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc8ac0683b0 0x7fc8ac06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.386+0000 7fc8cd775640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc8ac0683b0 0x7fc8ac06a870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80010f0 tx=0x7fc8b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:26.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.386+0000 7fc8be7fc640 1 -- 172.21.15.62:0/3692708758 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc8c40ac020 con 0x7fc8b40039c0 2024-04-18T10:06:26.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.550+0000 7fc8d01da640 1 -- 172.21.15.62:0/3692708758 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc8c80b1fd0 con 0x7fc8b40039c0 2024-04-18T10:06:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.550+0000 7fc8be7fc640 1 -- 172.21.15.62:0/3692708758 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fc8c4016780 con 0x7fc8b40039c0 2024-04-18T10:06:26.549 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-18T10:06:26.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.550+0000 7fc8d01da640 1 -- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc8ac0683b0 msgr2=0x7fc8ac06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.550+0000 7fc8d01da640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc8ac0683b0 0x7fc8ac06a870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc8b80010f0 tx=0x7fc8b8009040 comp rx=0 tx=0).stop 2024-04-18T10:06:26.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.550+0000 7fc8d01da640 1 -- 172.21.15.62:0/3692708758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 msgr2=0x7fc8c8122190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.550+0000 7fc8d01da640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 0x7fc8c8122190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009ae0 tx=0x7fc8c4019810 comp rx=0 tx=0).stop 2024-04-18T10:06:26.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.550+0000 7fc8d01da640 1 -- 172.21.15.62:0/3692708758 shutdown_connections 2024-04-18T10:06:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.550+0000 7fc8d01da640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc8ac0683b0 0x7fc8ac06a870 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.550+0000 7fc8d01da640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc8c81e1740 0x7fc8c8146310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.550+0000 7fc8d01da640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc8c809ecc0 0x7fc8c80b2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.550+0000 7fc8d01da640 1 --2- 172.21.15.62:0/3692708758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc8b40039c0 0x7fc8c8122190 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.550+0000 7fc8d01da640 1 -- 172.21.15.62:0/3692708758 >> 172.21.15.62:0/3692708758 conn(0x7fc8c80ba780 msgr2=0x7fc8c8148a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.554+0000 7fc8d01da640 1 -- 172.21.15.62:0/3692708758 shutdown_connections 2024-04-18T10:06:26.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.554+0000 7fc8d01da640 1 -- 172.21.15.62:0/3692708758 wait complete. 2024-04-18T10:06:26.568 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-18T10:06:26.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fc5ffa640 1 Processor -- start 2024-04-18T10:06:26.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fc5ffa640 1 -- start start 2024-04-18T10:06:26.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fc5ffa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fc006e610 0x7f4fc00afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fc5ffa640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4fc00b5650 0x7f4fc014f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fc5ffa640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fc00b0120 con 0x7f4fc00b0a80 2024-04-18T10:06:26.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fc5ffa640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f4fc00b6c10 con 0x7f4fc00b5650 2024-04-18T10:06:26.698 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fc5ffa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc00b6e60 con 0x7f4fc006e610 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbeffd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4fc00b0a80 0x7f4fc00b4f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35348/0 (socket says 172.21.15.62:35348) 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbeffd640 1 -- 172.21.15.62:0/1962927633 learned_addr learned my addr 172.21.15.62:0/1962927633 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbf7fe640 1 --2- 172.21.15.62:0/1962927633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fc006e610 0x7f4fc00afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbffff640 1 --2- 172.21.15.62:0/1962927633 >> v2:172.21.15.145:3300/0 conn(0x7f4fc00b5650 0x7f4fc014f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2569955334 0 0) 0x7f4fc00b0120 con 0x7f4fc00b0a80 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fa8005740 con 0x7f4fc00b0a80 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3313807346 0 0) 0x7f4fa8005740 con 0x7f4fc00b0a80 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fc006e610 msgr2=0x7f4fc00afbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 --2- 172.21.15.62:0/1962927633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fc006e610 0x7f4fc00afbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 >> v2:172.21.15.145:3300/0 conn(0x7f4fc00b5650 msgr2=0x7f4fc014f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 --2- 172.21.15.62:0/1962927633 >> v2:172.21.15.145:3300/0 conn(0x7f4fc00b5650 0x7f4fc014f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc006e9f0 con 0x7f4fc00b0a80 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f4fb4001680 con 0x7f4fc00b0a80 2024-04-18T10:06:26.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 >> v1:172.21.15.62:6789/0 conn(0x7f4fc00b0a80 legacy=0x7f4fc00b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 --2- 172.21.15.62:0/1962927633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fa8005bd0 0x7f4fa8005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 --2- 172.21.15.62:0/1962927633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fa8006a20 0x7f4fa8006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 --2- 172.21.15.62:0/1962927633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fc006e610 0x7f4fa80073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc00b6c10 con 0x7f4fa8005bd0 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc00b6e60 con 0x7f4fa8006a20 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f4fa800dc60 con 0x7f4fc006e610 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbf7fe640 1 --2- 172.21.15.62:0/1962927633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fa8005bd0 0x7f4fa8005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbffff640 1 --2- 172.21.15.62:0/1962927633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fc006e610 0x7f4fa80073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbeffd640 1 --2- 172.21.15.62:0/1962927633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fa8006a20 0x7f4fa8006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbf7fe640 1 -- 172.21.15.62:0/1962927633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fc006e610 msgr2=0x7f4fa80073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbf7fe640 1 --2- 172.21.15.62:0/1962927633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fc006e610 0x7f4fa80073e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbf7fe640 1 -- 172.21.15.62:0/1962927633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fa8006a20 msgr2=0x7f4fa8006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbf7fe640 1 --2- 172.21.15.62:0/1962927633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fa8006a20 0x7f4fa8006ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.698+0000 7f4fbf7fe640 1 -- 172.21.15.62:0/1962927633 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fa8005740 con 0x7f4fa8005bd0 2024-04-18T10:06:26.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbf7fe640 1 --2- 172.21.15.62:0/1962927633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fa8005bd0 0x7f4fa8005ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009ac0 tx=0x7f4fb001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1510716584894 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:26.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0009690 con 0x7f4fa8005bd0 2024-04-18T10:06:26.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fb00a6420 con 0x7f4fa8005bd0 2024-04-18T10:06:26.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbe7fc640 1 -- 172.21.15.62:0/1962927633 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb00a67f0 con 0x7f4fa8005bd0 2024-04-18T10:06:26.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/1962927633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fa8005bd0 msgr2=0x7f4fa8005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 --2- 172.21.15.62:0/1962927633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fa8005bd0 0x7f4fa8005ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0009ac0 tx=0x7f4fb001ca50 comp rx=0 tx=0).stop 2024-04-18T10:06:26.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbffff640 1 -- 172.21.15.62:0/1962927633 reap_dead start 2024-04-18T10:06:26.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/1962927633 shutdown_connections 2024-04-18T10:06:26.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/1962927633 >> 172.21.15.62:0/1962927633 conn(0x7f4fc00ba710 msgr2=0x7f4fc00bab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:26.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/1962927633 shutdown_connections 2024-04-18T10:06:26.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/1962927633 wait complete. 2024-04-18T10:06:26.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 Processor -- start 2024-04-18T10:06:26.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 -- start start 2024-04-18T10:06:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fa8005bd0 0x7f4fc0127d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fc0144640 0x7f4fc0144ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fc01e1160 0x7f4fc01e15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc009d090 con 0x7f4fc0144640 2024-04-18T10:06:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc009cdb0 con 0x7f4fc01e1160 2024-04-18T10:06:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fc5ffa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc009aa50 con 0x7f4fa8005bd0 2024-04-18T10:06:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbeffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fc0144640 0x7f4fc0144ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbeffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fc0144640 0x7f4fc0144ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37818/0 (socket says 172.21.15.62:37818) 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbeffd640 1 -- 172.21.15.62:0/2672369683 learned_addr learned my addr 172.21.15.62:0/2672369683 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbffff640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fc01e1160 0x7f4fc01e15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbf7fe640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fa8005bd0 0x7f4fc0127d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbeffd640 1 -- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fa8005bd0 msgr2=0x7f4fc0127d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbeffd640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fa8005bd0 0x7f4fc0127d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbeffd640 1 -- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fc01e1160 msgr2=0x7f4fc01e15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbeffd640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fc01e1160 0x7f4fc01e15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbeffd640 1 -- 172.21.15.62:0/2672369683 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc00ba2f0 con 0x7f4fc0144640 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.702+0000 7f4fbf7fe640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fa8005bd0 0x7f4fc0127d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.706+0000 7f4fbeffd640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fc0144640 0x7f4fc0144ac0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400d7a0 tx=0x7f4fb400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.706+0000 7f4fa7fff640 1 -- 172.21.15.62:0/2672369683 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4019070 con 0x7f4fc0144640 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.706+0000 7f4fa7fff640 1 -- 172.21.15.62:0/2672369683 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fb40072d0 con 0x7f4fc0144640 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.706+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/2672369683 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc00b45a0 con 0x7f4fc0144640 2024-04-18T10:06:26.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.706+0000 7f4fa7fff640 1 -- 172.21.15.62:0/2672369683 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb40047f0 con 0x7f4fc0144640 2024-04-18T10:06:26.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.706+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/2672369683 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc0127460 con 0x7f4fc0144640 2024-04-18T10:06:26.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.706+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/2672369683 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc0069230 con 0x7f4fc0144640 2024-04-18T10:06:26.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.706+0000 7f4fa7fff640 1 -- 172.21.15.62:0/2672369683 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4fb401c070 con 0x7f4fc0144640 2024-04-18T10:06:26.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.706+0000 7f4fa7fff640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4fac068380 0x7f4fac06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:26.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.706+0000 7f4fa7fff640 1 -- 172.21.15.62:0/2672369683 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4fb4086210 con 0x7f4fc0144640 2024-04-18T10:06:26.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.706+0000 7f4fbf7fe640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4fac068380 0x7f4fac06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:26.707 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.706+0000 7f4fbf7fe640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4fac068380 0x7f4fac06a840 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00ad000 tx=0x7f4fb00046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:26.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.710+0000 7f4fa7fff640 1 -- 172.21.15.62:0/2672369683 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4fb4056860 con 0x7f4fc0144640 2024-04-18T10:06:26.869 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.870+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/2672369683 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4fc009b590 con 0x7f4fc0144640 2024-04-18T10:06:26.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.870+0000 7f4fa7fff640 1 -- 172.21.15.62:0/2672369683 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f4fb4056160 con 0x7f4fc0144640 2024-04-18T10:06:26.870 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-04-18T10:06:26.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.874+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4fac068380 msgr2=0x7f4fac06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.874+0000 7f4fc5ffa640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4fac068380 0x7f4fac06a840 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00ad000 tx=0x7f4fb00046d0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.874+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/2672369683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fc0144640 msgr2=0x7f4fc0144ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:26.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.874+0000 7f4fc5ffa640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fc0144640 0x7f4fc0144ac0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400d7a0 tx=0x7f4fb400dc60 comp rx=0 tx=0).stop 2024-04-18T10:06:26.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.874+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/2672369683 shutdown_connections 2024-04-18T10:06:26.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.874+0000 7f4fc5ffa640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4fac068380 0x7f4fac06a840 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.874+0000 7f4fc5ffa640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fc01e1160 0x7f4fc01e15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.874+0000 7f4fc5ffa640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fc0144640 0x7f4fc0144ac0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.874+0000 7f4fc5ffa640 1 --2- 172.21.15.62:0/2672369683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fa8005bd0 0x7f4fc0127d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:26.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.874+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/2672369683 >> 172.21.15.62:0/2672369683 conn(0x7f4fc00ba710 msgr2=0x7f4fc0071f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:26.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.874+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/2672369683 shutdown_connections 2024-04-18T10:06:26.873 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:26.874+0000 7f4fc5ffa640 1 -- 172.21.15.62:0/2672369683 wait complete. 2024-04-18T10:06:26.888 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5232d7a640 1 Processor -- start 2024-04-18T10:06:27.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5232d7a640 1 -- start start 2024-04-18T10:06:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5232d7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c09be30 0x7f522c145f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5232d7a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f522c06d920 0x7f522c1504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5232d7a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f522c0ba170 con 0x7f522c0b9d90 2024-04-18T10:06:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5232d7a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f522c09cd30 con 0x7f522c06d920 2024-04-18T10:06:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5232d7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f522c146460 con 0x7f522c09be30 2024-04-18T10:06:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5231317640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f522c06d920 0x7f522c1504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5230b16640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f522c0b9d90 0x7f522c06d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35356/0 (socket says 172.21.15.62:35356) 2024-04-18T10:06:27.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5230b16640 1 -- 172.21.15.62:0/3809895113 learned_addr learned my addr 172.21.15.62:0/3809895113 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:27.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5231317640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f522c06d920 0x7f522c1504e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37822/0 (socket says 172.21.15.62:37822) 2024-04-18T10:06:27.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5231317640 1 -- 172.21.15.62:0/3809895113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c09be30 msgr2=0x7f522c145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:27.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5231317640 1 --2- 172.21.15.62:0/3809895113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c09be30 0x7f522c145f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5231317640 1 -- 172.21.15.62:0/3809895113 >> v1:172.21.15.62:6789/0 conn(0x7f522c0b9d90 legacy=0x7f522c06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f522b7fe640 1 -- 172.21.15.62:0/3809895113 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2412699938 0 0) 0x7f522c0ba170 con 0x7f522c0b9d90 2024-04-18T10:06:27.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.006+0000 7f5231317640 1 -- 172.21.15.62:0/3809895113 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522c14a060 con 0x7f522c06d920 2024-04-18T10:06:27.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.010+0000 7f5231317640 1 --2- 172.21.15.62:0/3809895113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522c06d920 0x7f522c1504e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5224009dc0 tx=0x7f522401c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3417842d19020e server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:27.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.010+0000 7f522b7fe640 1 -- 172.21.15.62:0/3809895113 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52240b2070 con 0x7f522c06d920 2024-04-18T10:06:27.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.010+0000 7f522b7fe640 1 -- 172.21.15.62:0/3809895113 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52240a57e0 con 0x7f522c06d920 2024-04-18T10:06:27.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.010+0000 7f522b7fe640 1 -- 172.21.15.62:0/3809895113 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52240a5bb0 con 0x7f522c06d920 2024-04-18T10:06:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.010+0000 7f5232d7a640 1 -- 172.21.15.62:0/3809895113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522c06d920 msgr2=0x7f522c1504e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.010+0000 7f5232d7a640 1 --2- 172.21.15.62:0/3809895113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522c06d920 0x7f522c1504e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5224009dc0 tx=0x7f522401c830 comp rx=0 tx=0).stop 2024-04-18T10:06:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.010+0000 7f5232d7a640 1 -- 172.21.15.62:0/3809895113 shutdown_connections 2024-04-18T10:06:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.010+0000 7f5232d7a640 1 --2- 172.21.15.62:0/3809895113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522c06d920 0x7f522c1504e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.010+0000 7f5232d7a640 1 --2- 172.21.15.62:0/3809895113 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c09be30 0x7f522c145f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.010+0000 7f5232d7a640 1 -- 172.21.15.62:0/3809895113 >> 172.21.15.62:0/3809895113 conn(0x7f522c06c0d0 msgr2=0x7f522c0b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.010+0000 7f5232d7a640 1 -- 172.21.15.62:0/3809895113 shutdown_connections 2024-04-18T10:06:27.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.010+0000 7f5232d7a640 1 -- 172.21.15.62:0/3809895113 wait complete. 2024-04-18T10:06:27.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f5232d7a640 1 Processor -- start 2024-04-18T10:06:27.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f5232d7a640 1 -- start start 2024-04-18T10:06:27.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f5232d7a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c06d920 0x7f522c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f5232d7a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522c09be30 0x7f522c13fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f5232d7a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f522c0b9d90 0x7f522c140690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f5232d7a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f522c0b3fa0 con 0x7f522c09be30 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f5232d7a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f522c0b3e20 con 0x7f522c0b9d90 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f5232d7a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f522c153240 con 0x7f522c06d920 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f522bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522c09be30 0x7f522c13fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f522bfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522c09be30 0x7f522c13fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37838/0 (socket says 172.21.15.62:37838) 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f522bfff640 1 -- 172.21.15.62:0/1785835173 learned_addr learned my addr 172.21.15.62:0/1785835173 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f522bfff640 1 -- 172.21.15.62:0/1785835173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c06d920 msgr2=0x7f522c0b8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f522bfff640 1 --2- 172.21.15.62:0/1785835173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c06d920 0x7f522c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f522bfff640 1 -- 172.21.15.62:0/1785835173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f522c0b9d90 msgr2=0x7f522c140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f522bfff640 1 --2- 172.21.15.62:0/1785835173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f522c0b9d90 0x7f522c140690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f522bfff640 1 -- 172.21.15.62:0/1785835173 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522c13d5a0 con 0x7f522c09be30 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f522bfff640 1 --2- 172.21.15.62:0/1785835173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522c09be30 0x7f522c13fdb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f522000b580 tx=0x7f522000ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f52297fa640 1 -- 172.21.15.62:0/1785835173 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220014070 con 0x7f522c09be30 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f52297fa640 1 -- 172.21.15.62:0/1785835173 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5220010890 con 0x7f522c09be30 2024-04-18T10:06:27.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f52297fa640 1 -- 172.21.15.62:0/1785835173 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5220010c80 con 0x7f522c09be30 2024-04-18T10:06:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f5232d7a640 1 -- 172.21.15.62:0/1785835173 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522c13d840 con 0x7f522c09be30 2024-04-18T10:06:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f5232d7a640 1 -- 172.21.15.62:0/1785835173 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f522c14f780 con 0x7f522c09be30 2024-04-18T10:06:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f5232d7a640 1 -- 172.21.15.62:0/1785835173 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f522c068b90 con 0x7f522c09be30 2024-04-18T10:06:27.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f52297fa640 1 -- 172.21.15.62:0/1785835173 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5220017070 con 0x7f522c09be30 2024-04-18T10:06:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f52297fa640 1 --2- 172.21.15.62:0/1785835173 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5218068380 0x7f521806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f52297fa640 1 -- 172.21.15.62:0/1785835173 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5220086780 con 0x7f522c09be30 2024-04-18T10:06:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.014+0000 7f5230b16640 1 --2- 172.21.15.62:0/1785835173 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5218068380 0x7f521806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.018+0000 7f5230b16640 1 --2- 172.21.15.62:0/1785835173 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5218068380 0x7f521806a840 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f521c0015a0 tx=0x7f521c007970 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:27.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.018+0000 7f52297fa640 1 -- 172.21.15.62:0/1785835173 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5220056d50 con 0x7f522c09be30 2024-04-18T10:06:27.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 -- 172.21.15.62:0/1785835173 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f522c0b0160 con 0x7f522c09be30 2024-04-18T10:06:27.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f52297fa640 1 -- 172.21.15.62:0/1785835173 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3527 (secure 0 0 0) 0x7f5220010e20 con 0x7f522c09be30 2024-04-18T10:06:27.209 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:27.209 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/904929673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":904929673},{"type":"v1","addr":"172.21.15.145:6835","nonce":904929673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:05:58.588100+0000","modified":"2024-04-18T10:06:01.605447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2120622858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2120622858},{"type":"v1","addr":"172.21.15.62:6833","nonce":2120622858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-18T10:06:27.209 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-04-18T10:06:27.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 -- 172.21.15.62:0/1785835173 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5218068380 msgr2=0x7f521806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 --2- 172.21.15.62:0/1785835173 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5218068380 0x7f521806a840 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f521c0015a0 tx=0x7f521c007970 comp rx=0 tx=0).stop 2024-04-18T10:06:27.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 -- 172.21.15.62:0/1785835173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522c09be30 msgr2=0x7f522c13fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 --2- 172.21.15.62:0/1785835173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522c09be30 0x7f522c13fdb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f522000b580 tx=0x7f522000ba40 comp rx=0 tx=0).stop 2024-04-18T10:06:27.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 -- 172.21.15.62:0/1785835173 shutdown_connections 2024-04-18T10:06:27.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 --2- 172.21.15.62:0/1785835173 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5218068380 0x7f521806a840 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 --2- 172.21.15.62:0/1785835173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f522c0b9d90 0x7f522c140690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 --2- 172.21.15.62:0/1785835173 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f522c09be30 0x7f522c13fdb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 --2- 172.21.15.62:0/1785835173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c06d920 0x7f522c0b8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 -- 172.21.15.62:0/1785835173 >> 172.21.15.62:0/1785835173 conn(0x7f522c06c0d0 msgr2=0x7f522c0b6670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:27.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 -- 172.21.15.62:0/1785835173 shutdown_connections 2024-04-18T10:06:27.211 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.210+0000 7f5232d7a640 1 -- 172.21.15.62:0/1785835173 wait complete. 2024-04-18T10:06:27.226 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f659b5a6640 1 Processor -- start 2024-04-18T10:06:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f659b5a6640 1 -- start start 2024-04-18T10:06:27.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f659b5a6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940b9d90 0x7f659406d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f659b5a6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f659409be30 0x7f659406d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f659b5a6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65940ba170 con 0x7f659406dc90 2024-04-18T10:06:27.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f659b5a6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f659409cd30 con 0x7f659409be30 2024-04-18T10:06:27.349 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f659b5a6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f659414a060 con 0x7f65940b9d90 2024-04-18T10:06:27.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f6599342640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940b9d90 0x7f659406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f6598b41640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f659409be30 0x7f659406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f6598b41640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f659409be30 0x7f659406d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.62:35540/0 (socket says 172.21.15.62:35540) 2024-04-18T10:06:27.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f6598b41640 1 -- 172.21.15.62:0/1292982351 learned_addr learned my addr 172.21.15.62:0/1292982351 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:27.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f6599342640 1 -- 172.21.15.62:0/1292982351 >> v2:172.21.15.145:3300/0 conn(0x7f659409be30 msgr2=0x7f659406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:27.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f6599342640 1 --2- 172.21.15.62:0/1292982351 >> v2:172.21.15.145:3300/0 conn(0x7f659409be30 0x7f659406d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f6599342640 1 -- 172.21.15.62:0/1292982351 >> v1:172.21.15.145:6789/0 conn(0x7f659406dc90 legacy=0x7f6594144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.350+0000 7f6599342640 1 -- 172.21.15.62:0/1292982351 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6594149860 con 0x7f65940b9d90 2024-04-18T10:06:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f6599342640 1 --2- 172.21.15.62:0/1292982351 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940b9d90 0x7f659406d210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f65900097c0 tx=0x7f659001c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0218581f1048d60 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f658bfff640 1 -- 172.21.15.62:0/1292982351 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65900b0070 con 0x7f65940b9d90 2024-04-18T10:06:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f658bfff640 1 -- 172.21.15.62:0/1292982351 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65900ab600 con 0x7f65940b9d90 2024-04-18T10:06:27.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f658bfff640 1 -- 172.21.15.62:0/1292982351 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65900ab9d0 con 0x7f65940b9d90 2024-04-18T10:06:27.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f659b5a6640 1 -- 172.21.15.62:0/1292982351 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940b9d90 msgr2=0x7f659406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f659b5a6640 1 --2- 172.21.15.62:0/1292982351 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940b9d90 0x7f659406d210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f65900097c0 tx=0x7f659001c380 comp rx=0 tx=0).stop 2024-04-18T10:06:27.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f659b5a6640 1 -- 172.21.15.62:0/1292982351 shutdown_connections 2024-04-18T10:06:27.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f659b5a6640 1 --2- 172.21.15.62:0/1292982351 >> v2:172.21.15.145:3300/0 conn(0x7f659409be30 0x7f659406d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f659b5a6640 1 --2- 172.21.15.62:0/1292982351 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940b9d90 0x7f659406d210 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f659b5a6640 1 -- 172.21.15.62:0/1292982351 >> 172.21.15.62:0/1292982351 conn(0x7f659406c0d0 msgr2=0x7f65940b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f659b5a6640 1 -- 172.21.15.62:0/1292982351 shutdown_connections 2024-04-18T10:06:27.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f659b5a6640 1 -- 172.21.15.62:0/1292982351 wait complete. 2024-04-18T10:06:27.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f659b5a6640 1 Processor -- start 2024-04-18T10:06:27.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.354+0000 7f659b5a6640 1 -- start start 2024-04-18T10:06:27.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f659b5a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f659406dc90 0x7f65941e0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f659b5a6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f659409be30 0x7f65940b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f659b5a6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940b9d90 0x7f65941501d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f659b5a6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6594145fb0 con 0x7f659406dc90 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f659b5a6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f6594145e30 con 0x7f659409be30 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f659b5a6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f6594146130 con 0x7f65940b9d90 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6599342640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f659406dc90 0x7f65941e0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6599342640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f659406dc90 0x7f65941e0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37846/0 (socket says 172.21.15.62:37846) 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6599342640 1 -- 172.21.15.62:0/829389559 learned_addr learned my addr 172.21.15.62:0/829389559 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6598b41640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f659409be30 0x7f65940b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6599b43640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940b9d90 0x7f65941501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6599342640 1 -- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940b9d90 msgr2=0x7f65941501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6599342640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940b9d90 0x7f65941501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6599342640 1 -- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f659409be30 msgr2=0x7f65940b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6599342640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f659409be30 0x7f65940b31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6599342640 1 -- 172.21.15.62:0/829389559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f659413f370 con 0x7f659406dc90 2024-04-18T10:06:27.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6599342640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f659406dc90 0x7f65941e0510 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f65900ae040 tx=0x7f659001abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:27.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6589ffb640 1 -- 172.21.15.62:0/829389559 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65900b0050 con 0x7f659406dc90 2024-04-18T10:06:27.358 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6589ffb640 1 -- 172.21.15.62:0/829389559 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65900b1040 con 0x7f659406dc90 2024-04-18T10:06:27.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f659b5a6640 1 -- 172.21.15.62:0/829389559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f659413f650 con 0x7f659406dc90 2024-04-18T10:06:27.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f6589ffb640 1 -- 172.21.15.62:0/829389559 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6590006e20 con 0x7f659406dc90 2024-04-18T10:06:27.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.358+0000 7f659b5a6640 1 -- 172.21.15.62:0/829389559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f659413e1b0 con 0x7f659406dc90 2024-04-18T10:06:27.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.362+0000 7f659b5a6640 1 -- 172.21.15.62:0/829389559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6594068b90 con 0x7f659406dc90 2024-04-18T10:06:27.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.362+0000 7f6589ffb640 1 -- 172.21.15.62:0/829389559 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f65900be030 con 0x7f659406dc90 2024-04-18T10:06:27.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.362+0000 7f6589ffb640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6578068380 0x7f657806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.362+0000 7f6589ffb640 1 -- 172.21.15.62:0/829389559 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6590120660 con 0x7f659406dc90 2024-04-18T10:06:27.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.362+0000 7f6598b41640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6578068380 0x7f657806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.362+0000 7f6598b41640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6578068380 0x7f657806a840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6594150a80 tx=0x7f6584008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:27.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.366+0000 7f6589ffb640 1 -- 172.21.15.62:0/829389559 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f65900c5060 con 0x7f659406dc90 2024-04-18T10:06:27.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.530+0000 7f659b5a6640 1 -- 172.21.15.62:0/829389559 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65940af380 con 0x7f659406dc90 2024-04-18T10:06:27.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f6589ffb640 1 -- 172.21.15.62:0/829389559 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f6590016090 con 0x7f659406dc90 2024-04-18T10:06:27.532 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:27.533 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f659b5a6640 1 -- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6578068380 msgr2=0x7f657806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f659b5a6640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6578068380 0x7f657806a840 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6594150a80 tx=0x7f6584008040 comp rx=0 tx=0).stop 2024-04-18T10:06:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f659b5a6640 1 -- 172.21.15.62:0/829389559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f659406dc90 msgr2=0x7f65941e0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f659b5a6640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f659406dc90 0x7f65941e0510 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f65900ae040 tx=0x7f659001abc0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f659b5a6640 1 -- 172.21.15.62:0/829389559 shutdown_connections 2024-04-18T10:06:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f659b5a6640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6578068380 0x7f657806a840 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f659b5a6640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65940b9d90 0x7f65941501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f659b5a6640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f659409be30 0x7f65940b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f659b5a6640 1 --2- 172.21.15.62:0/829389559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f659406dc90 0x7f65941e0510 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f659b5a6640 1 -- 172.21.15.62:0/829389559 >> 172.21.15.62:0/829389559 conn(0x7f659406c0d0 msgr2=0x7f659406d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f659b5a6640 1 -- 172.21.15.62:0/829389559 shutdown_connections 2024-04-18T10:06:27.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.534+0000 7f659b5a6640 1 -- 172.21.15.62:0/829389559 wait complete. 2024-04-18T10:06:27.552 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-18T10:06:27.553 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:27.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.670+0000 7f275da88640 1 Processor -- start 2024-04-18T10:06:27.671 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.670+0000 7f275da88640 1 -- start start 2024-04-18T10:06:27.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f275da88640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f27580b5420 0x7f27580b5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f275da88640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27580b5dd0 con 0x7f27580b6260 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f275da88640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27580b7850 con 0x7f27580b6640 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f275da88640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f2758148cb0 con 0x7f27580b5420 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27577fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f27580b6260 0x7f2758148260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35362/0 (socket says 172.21.15.62:35362) 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27577fe640 1 -- 172.21.15.62:0/2535801511 learned_addr learned my addr 172.21.15.62:0/2535801511 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2702675049 0 0) 0x7f27580b5dd0 con 0x7f27580b6260 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f2757fff640 1 --2- 172.21.15.62:0/2535801511 >> v2:172.21.15.145:3300/0 conn(0x7f27580b5420 0x7f27580b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2740003530 con 0x7f27580b6260 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1344038172 0 0) 0x7f2740003530 con 0x7f27580b6260 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 >> v2:172.21.15.145:3300/0 conn(0x7f27580b5420 msgr2=0x7f27580b5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 --2- 172.21.15.62:0/2535801511 >> v2:172.21.15.145:3300/0 conn(0x7f27580b5420 0x7f27580b5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 >> v1:172.21.15.145:6789/0 conn(0x7f27580b6640 legacy=0x7f27581419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27580b4c20 con 0x7f27580b6260 2024-04-18T10:06:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f2748001680 con 0x7f27580b6260 2024-04-18T10:06:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 >> v1:172.21.15.62:6789/0 conn(0x7f27580b6260 legacy=0x7f2758148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 --2- 172.21.15.62:0/2535801511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27400039c0 0x7f2740003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 --2- 172.21.15.62:0/2535801511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2740004810 0x7f2740004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 --2- 172.21.15.62:0/2535801511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27400052f0 0x7f274000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2758148cb0 con 0x7f27400039c0 2024-04-18T10:06:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f274000c1c0 con 0x7f2740004810 2024-04-18T10:06:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f274000c3c0 con 0x7f27400052f0 2024-04-18T10:06:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27577fe640 1 --2- 172.21.15.62:0/2535801511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27400039c0 0x7f2740003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f2756ffd640 1 --2- 172.21.15.62:0/2535801511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2740004810 0x7f2740004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f2757fff640 1 --2- 172.21.15.62:0/2535801511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27400052f0 0x7f274000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27577fe640 1 -- 172.21.15.62:0/2535801511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27400052f0 msgr2=0x7f274000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:27.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27577fe640 1 --2- 172.21.15.62:0/2535801511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27400052f0 0x7f274000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27577fe640 1 -- 172.21.15.62:0/2535801511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2740004810 msgr2=0x7f2740004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:27.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27577fe640 1 --2- 172.21.15.62:0/2535801511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2740004810 0x7f2740004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f27577fe640 1 -- 172.21.15.62:0/2535801511 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2740003530 con 0x7f27400039c0 2024-04-18T10:06:27.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f2757fff640 1 -- 172.21.15.62:0/2535801511 reap_dead start 2024-04-18T10:06:27.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.674+0000 7f2756ffd640 1 --2- 172.21.15.62:0/2535801511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2740004810 0x7f2740004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:27.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f27577fe640 1 --2- 172.21.15.62:0/2535801511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27400039c0 0x7f2740003de0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f274800c190 tx=0x7f274801cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b793d083b47be43 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:27.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2748007d00 con 0x7f27400039c0 2024-04-18T10:06:27.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27480a9020 con 0x7f27400039c0 2024-04-18T10:06:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f27567fc640 1 -- 172.21.15.62:0/2535801511 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2758148a70 con 0x7f27400039c0 2024-04-18T10:06:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 -- 172.21.15.62:0/2535801511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27400039c0 msgr2=0x7f2740003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 --2- 172.21.15.62:0/2535801511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27400039c0 0x7f2740003de0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f274800c190 tx=0x7f274801cba0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 -- 172.21.15.62:0/2535801511 shutdown_connections 2024-04-18T10:06:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 --2- 172.21.15.62:0/2535801511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27400039c0 0x7f2740003de0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 -- 172.21.15.62:0/2535801511 >> 172.21.15.62:0/2535801511 conn(0x7f27580bd5c0 msgr2=0x7f27580bd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 -- 172.21.15.62:0/2535801511 shutdown_connections 2024-04-18T10:06:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 -- 172.21.15.62:0/2535801511 wait complete. 2024-04-18T10:06:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 Processor -- start 2024-04-18T10:06:27.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 -- start start 2024-04-18T10:06:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27400039c0 0x7f27581d56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27581d4520 0x7f27581e0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27581e0c40 0x7f27581e3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2758141d30 con 0x7f27581e0c40 2024-04-18T10:06:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f2758141bb0 con 0x7f27400039c0 2024-04-18T10:06:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f275da88640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2758141eb0 con 0x7f27581d4520 2024-04-18T10:06:27.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f2757fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27581e0c40 0x7f27581e3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f2757fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27581e0c40 0x7f27581e3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37860/0 (socket says 172.21.15.62:37860) 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f2757fff640 1 -- 172.21.15.62:0/2546387056 learned_addr learned my addr 172.21.15.62:0/2546387056 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f27577fe640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27400039c0 0x7f27581d56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f2756ffd640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27581d4520 0x7f27581e0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f2757fff640 1 -- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27581d4520 msgr2=0x7f27581e0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f2757fff640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27581d4520 0x7f27581e0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f2757fff640 1 -- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27400039c0 msgr2=0x7f27581d56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f2757fff640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27400039c0 0x7f27581d56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f2757fff640 1 -- 172.21.15.62:0/2546387056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2758142fd0 con 0x7f27581e0c40 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f2757fff640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27581e0c40 0x7f27581e3700 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f27580b76e0 tx=0x7f27500115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f273ffff640 1 -- 172.21.15.62:0/2546387056 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2750011e10 con 0x7f27581e0c40 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f273ffff640 1 -- 172.21.15.62:0/2546387056 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f275001b070 con 0x7f27581e0c40 2024-04-18T10:06:27.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.678+0000 7f273ffff640 1 -- 172.21.15.62:0/2546387056 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27500039a0 con 0x7f27581e0c40 2024-04-18T10:06:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.682+0000 7f275da88640 1 -- 172.21.15.62:0/2546387056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27581d5cc0 con 0x7f27581e0c40 2024-04-18T10:06:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.682+0000 7f275da88640 1 -- 172.21.15.62:0/2546387056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27581d3700 con 0x7f27581e0c40 2024-04-18T10:06:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.682+0000 7f275da88640 1 -- 172.21.15.62:0/2546387056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2758073bf0 con 0x7f27581e0c40 2024-04-18T10:06:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.682+0000 7f273ffff640 1 -- 172.21.15.62:0/2546387056 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2750003b40 con 0x7f27581e0c40 2024-04-18T10:06:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.682+0000 7f273ffff640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2744068380 0x7f274406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.682+0000 7f273ffff640 1 -- 172.21.15.62:0/2546387056 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2750087100 con 0x7f27581e0c40 2024-04-18T10:06:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.682+0000 7f27577fe640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2744068380 0x7f274406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:27.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.682+0000 7f27577fe640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2744068380 0x7f274406a840 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f27480aa000 tx=0x7f27580b7a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:27.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.686+0000 7f273ffff640 1 -- 172.21.15.62:0/2546387056 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f27500576a0 con 0x7f27581e0c40 2024-04-18T10:06:27.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.878+0000 7f275da88640 1 -- 172.21.15.62:0/2546387056 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2758068b90 con 0x7f27581e0c40 2024-04-18T10:06:27.879 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f273ffff640 1 -- 172.21.15.62:0/2546387056 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3527 (secure 0 0 0) 0x7f2750056fa0 con 0x7f27581e0c40 2024-04-18T10:06:27.880 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:27.880 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/904929673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":904929673},{"type":"v1","addr":"172.21.15.145:6835","nonce":904929673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:05:58.588100+0000","modified":"2024-04-18T10:06:01.605447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2120622858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2120622858},{"type":"v1","addr":"172.21.15.62:6833","nonce":2120622858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-18T10:06:27.880 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-04-18T10:06:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f275da88640 1 -- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2744068380 msgr2=0x7f274406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f275da88640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2744068380 0x7f274406a840 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f27480aa000 tx=0x7f27580b7a20 comp rx=0 tx=0).stop 2024-04-18T10:06:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f275da88640 1 -- 172.21.15.62:0/2546387056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27581e0c40 msgr2=0x7f27581e3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f275da88640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27581e0c40 0x7f27581e3700 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f27580b76e0 tx=0x7f27500115a0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f275da88640 1 -- 172.21.15.62:0/2546387056 shutdown_connections 2024-04-18T10:06:27.882 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f275da88640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2744068380 0x7f274406a840 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f275da88640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f27581e0c40 0x7f27581e3700 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f275da88640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27581d4520 0x7f27581e0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f275da88640 1 --2- 172.21.15.62:0/2546387056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27400039c0 0x7f27581d56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:27.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f275da88640 1 -- 172.21.15.62:0/2546387056 >> 172.21.15.62:0/2546387056 conn(0x7f27580bd5c0 msgr2=0x7f27580b9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:27.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f275da88640 1 -- 172.21.15.62:0/2546387056 shutdown_connections 2024-04-18T10:06:27.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:27.882+0000 7f275da88640 1 -- 172.21.15.62:0/2546387056 wait complete. 2024-04-18T10:06:27.897 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:28.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d97b86640 1 Processor -- start 2024-04-18T10:06:28.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d97b86640 1 -- start start 2024-04-18T10:06:28.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d97b86640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0d900b9d90 0x7f0d9006d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d97b86640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d9009be30 0x7f0d9006d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d97b86640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d900ba170 con 0x7f0d9006dc90 2024-04-18T10:06:28.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d97b86640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f0d9009cd30 con 0x7f0d900b9d90 2024-04-18T10:06:28.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d97b86640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f0d9014a060 con 0x7f0d9009be30 2024-04-18T10:06:28.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d95922640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0d900b9d90 0x7f0d9006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d95922640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0d900b9d90 0x7f0d9006d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37864/0 (socket says 172.21.15.62:37864) 2024-04-18T10:06:28.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d95922640 1 -- 172.21.15.62:0/1054747600 learned_addr learned my addr 172.21.15.62:0/1054747600 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:28.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d95922640 1 -- 172.21.15.62:0/1054747600 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d9009be30 msgr2=0x7f0d9006d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:28.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d95922640 1 --2- 172.21.15.62:0/1054747600 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d9009be30 0x7f0d9006d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d95922640 1 -- 172.21.15.62:0/1054747600 >> v1:172.21.15.145:6789/0 conn(0x7f0d9006dc90 legacy=0x7f0d90144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:28.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.010+0000 7f0d95922640 1 -- 172.21.15.62:0/1054747600 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d90149860 con 0x7f0d900b9d90 2024-04-18T10:06:28.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d95922640 1 --2- 172.21.15.62:0/1054747600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d900b9d90 0x7f0d9006d210 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009870 tx=0x7f0d8c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9bde8e3df2a542 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:28.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d94920640 1 -- 172.21.15.62:0/1054747600 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0b0070 con 0x7f0d900b9d90 2024-04-18T10:06:28.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d94920640 1 -- 172.21.15.62:0/1054747600 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d8c0ab600 con 0x7f0d900b9d90 2024-04-18T10:06:28.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d94920640 1 -- 172.21.15.62:0/1054747600 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0ab9d0 con 0x7f0d900b9d90 2024-04-18T10:06:28.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d97b86640 1 -- 172.21.15.62:0/1054747600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d900b9d90 msgr2=0x7f0d9006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d97b86640 1 --2- 172.21.15.62:0/1054747600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d900b9d90 0x7f0d9006d210 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c009870 tx=0x7f0d8c01c400 comp rx=0 tx=0).stop 2024-04-18T10:06:28.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d97b86640 1 -- 172.21.15.62:0/1054747600 shutdown_connections 2024-04-18T10:06:28.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d97b86640 1 --2- 172.21.15.62:0/1054747600 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d9009be30 0x7f0d9006d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d97b86640 1 --2- 172.21.15.62:0/1054747600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d900b9d90 0x7f0d9006d210 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d97b86640 1 -- 172.21.15.62:0/1054747600 >> 172.21.15.62:0/1054747600 conn(0x7f0d9006c0d0 msgr2=0x7f0d900b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:28.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d97b86640 1 -- 172.21.15.62:0/1054747600 shutdown_connections 2024-04-18T10:06:28.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d97b86640 1 -- 172.21.15.62:0/1054747600 wait complete. 2024-04-18T10:06:28.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d97b86640 1 Processor -- start 2024-04-18T10:06:28.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.014+0000 7f0d97b86640 1 -- start start 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d97b86640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d9006dc90 0x7f0d90143480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d97b86640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d9009be30 0x7f0d900b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d97b86640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900b9d90 0x7f0d9013fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d97b86640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0d90145fb0 con 0x7f0d9006dc90 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d97b86640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f0d90145e30 con 0x7f0d900b9d90 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d97b86640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f0d90146130 con 0x7f0d9009be30 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d95922640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d9006dc90 0x7f0d90143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d95922640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d9006dc90 0x7f0d90143480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37868/0 (socket says 172.21.15.62:37868) 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d95922640 1 -- 172.21.15.62:0/1334075650 learned_addr learned my addr 172.21.15.62:0/1334075650 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d95121640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d9009be30 0x7f0d900b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d96123640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900b9d90 0x7f0d9013fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d95922640 1 -- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d9009be30 msgr2=0x7f0d900b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d95922640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d9009be30 0x7f0d900b0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d95922640 1 -- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900b9d90 msgr2=0x7f0d9013fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d95922640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900b9d90 0x7f0d9013fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d95922640 1 -- 172.21.15.62:0/1334075650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9013cc60 con 0x7f0d9006dc90 2024-04-18T10:06:28.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d96123640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900b9d90 0x7f0d9013fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:28.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d95922640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d9006dc90 0x7f0d90143480 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c0ae040 tx=0x7f0d8c01ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:28.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d867fc640 1 -- 172.21.15.62:0/1334075650 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0ba020 con 0x7f0d9006dc90 2024-04-18T10:06:28.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d867fc640 1 -- 172.21.15.62:0/1334075650 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d8c0b6620 con 0x7f0d9006dc90 2024-04-18T10:06:28.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d97b86640 1 -- 172.21.15.62:0/1334075650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d900b4cd0 con 0x7f0d9006dc90 2024-04-18T10:06:28.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d867fc640 1 -- 172.21.15.62:0/1334075650 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0b0040 con 0x7f0d9006dc90 2024-04-18T10:06:28.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d97b86640 1 -- 172.21.15.62:0/1334075650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d9013f460 con 0x7f0d9006dc90 2024-04-18T10:06:28.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d97b86640 1 -- 172.21.15.62:0/1334075650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d90068b90 con 0x7f0d9006dc90 2024-04-18T10:06:28.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d867fc640 1 -- 172.21.15.62:0/1334075650 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0d8c0bd020 con 0x7f0d9006dc90 2024-04-18T10:06:28.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d867fc640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0d74068380 0x7f0d7406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d867fc640 1 -- 172.21.15.62:0/1334075650 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0d8c120b90 con 0x7f0d9006dc90 2024-04-18T10:06:28.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.018+0000 7f0d95121640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0d74068380 0x7f0d7406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.022+0000 7f0d95121640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0d74068380 0x7f0d7406a840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0d80006fd0 tx=0x7f0d80008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:28.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.022+0000 7f0d867fc640 1 -- 172.21.15.62:0/1334075650 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0d8c0c4030 con 0x7f0d9006dc90 2024-04-18T10:06:28.215 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.214+0000 7f0d97b86640 1 -- 172.21.15.62:0/1334075650 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d9006cae0 con 0x7f0d9006dc90 2024-04-18T10:06:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d867fc640 1 -- 172.21.15.62:0/1334075650 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+3527 (secure 0 0 0) 0x7f0d8c016110 con 0x7f0d9006dc90 2024-04-18T10:06:28.216 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:28.216 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/904929673","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":904929673},{"type":"v1","addr":"172.21.15.145:6835","nonce":904929673}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:05:58.588100+0000","modified":"2024-04-18T10:06:01.605447+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4251},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4251":{"gid":4251,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2120622858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2120622858},{"type":"v1","addr":"172.21.15.62:6833","nonce":2120622858}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4251,"qdb_cluster":[4251]},"id":1}]} 2024-04-18T10:06:28.216 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 8 2024-04-18T10:06:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d97b86640 1 -- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0d74068380 msgr2=0x7f0d7406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d97b86640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0d74068380 0x7f0d7406a840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0d80006fd0 tx=0x7f0d80008040 comp rx=0 tx=0).stop 2024-04-18T10:06:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d97b86640 1 -- 172.21.15.62:0/1334075650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d9006dc90 msgr2=0x7f0d90143480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d97b86640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d9006dc90 0x7f0d90143480 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c0ae040 tx=0x7f0d8c01ac40 comp rx=0 tx=0).stop 2024-04-18T10:06:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d97b86640 1 -- 172.21.15.62:0/1334075650 shutdown_connections 2024-04-18T10:06:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d97b86640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f0d74068380 0x7f0d7406a840 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.218 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d97b86640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0d900b9d90 0x7f0d9013fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d97b86640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0d9009be30 0x7f0d900b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d97b86640 1 --2- 172.21.15.62:0/1334075650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d9006dc90 0x7f0d90143480 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d97b86640 1 -- 172.21.15.62:0/1334075650 >> 172.21.15.62:0/1334075650 conn(0x7f0d9006c0d0 msgr2=0x7f0d900ae650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d97b86640 1 -- 172.21.15.62:0/1334075650 shutdown_connections 2024-04-18T10:06:28.219 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.218+0000 7f0d97b86640 1 -- 172.21.15.62:0/1334075650 wait complete. 2024-04-18T10:06:28.233 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:28.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9287465640 1 Processor -- start 2024-04-18T10:06:28.351 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9287465640 1 -- start start 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9287465640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f92800b9d90 0x7f928006d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9287465640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9280145f20 0x7f92801504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9287465640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92800ba170 con 0x7f928009be30 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9287465640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f928009cd30 con 0x7f92800b9d90 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9287465640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f928014a060 con 0x7f9280145f20 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9285201640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f92800b9d90 0x7f928006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9284a00640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f928009be30 0x7f928006d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35376/0 (socket says 172.21.15.62:35376) 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9284a00640 1 -- 172.21.15.62:0/450012412 learned_addr learned my addr 172.21.15.62:0/450012412 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9285a02640 1 --2- 172.21.15.62:0/450012412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9280145f20 0x7f92801504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9285201640 1 -- 172.21.15.62:0/450012412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9280145f20 msgr2=0x7f92801504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9285201640 1 --2- 172.21.15.62:0/450012412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9280145f20 0x7f92801504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9285201640 1 -- 172.21.15.62:0/450012412 >> v1:172.21.15.62:6789/0 conn(0x7f928009be30 legacy=0x7f928006d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9277fff640 1 -- 172.21.15.62:0/450012412 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4066574346 0 0) 0x7f92800ba170 con 0x7f928009be30 2024-04-18T10:06:28.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9285201640 1 -- 172.21.15.62:0/450012412 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9280149860 con 0x7f92800b9d90 2024-04-18T10:06:28.354 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9285a02640 1 --2- 172.21.15.62:0/450012412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9280145f20 0x7f92801504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:28.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9285201640 1 --2- 172.21.15.62:0/450012412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92800b9d90 0x7f928006d210 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f927c009d90 tx=0x7f927c01c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa6f1d0f40131b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:28.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.354+0000 7f9277fff640 1 -- 172.21.15.62:0/450012412 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927c0b2070 con 0x7f92800b9d90 2024-04-18T10:06:28.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9277fff640 1 -- 172.21.15.62:0/450012412 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f927c0a57e0 con 0x7f92800b9d90 2024-04-18T10:06:28.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9277fff640 1 -- 172.21.15.62:0/450012412 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927c0a5bb0 con 0x7f92800b9d90 2024-04-18T10:06:28.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 -- 172.21.15.62:0/450012412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92800b9d90 msgr2=0x7f928006d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 --2- 172.21.15.62:0/450012412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92800b9d90 0x7f928006d210 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f927c009d90 tx=0x7f927c01c850 comp rx=0 tx=0).stop 2024-04-18T10:06:28.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 -- 172.21.15.62:0/450012412 shutdown_connections 2024-04-18T10:06:28.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 --2- 172.21.15.62:0/450012412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9280145f20 0x7f92801504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 --2- 172.21.15.62:0/450012412 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92800b9d90 0x7f928006d210 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 -- 172.21.15.62:0/450012412 >> 172.21.15.62:0/450012412 conn(0x7f928006c0d0 msgr2=0x7f92800b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:28.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 -- 172.21.15.62:0/450012412 shutdown_connections 2024-04-18T10:06:28.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 -- 172.21.15.62:0/450012412 wait complete. 2024-04-18T10:06:28.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 Processor -- start 2024-04-18T10:06:28.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 -- start start 2024-04-18T10:06:28.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f928009be30 0x7f9280142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92800b9d90 0x7f9280147af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9280145f20 0x7f92801e33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f92800b3fa0 con 0x7f92800b9d90 2024-04-18T10:06:28.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f92800b3e20 con 0x7f9280145f20 2024-04-18T10:06:28.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f928006de60 con 0x7f928009be30 2024-04-18T10:06:28.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9284a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92800b9d90 0x7f9280147af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9284a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92800b9d90 0x7f9280147af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37880/0 (socket says 172.21.15.62:37880) 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9284a00640 1 -- 172.21.15.62:0/2206307753 learned_addr learned my addr 172.21.15.62:0/2206307753 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9285201640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f928009be30 0x7f9280142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9285a02640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9280145f20 0x7f92801e33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9284a00640 1 -- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f928009be30 msgr2=0x7f9280142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9284a00640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f928009be30 0x7f9280142b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9284a00640 1 -- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9280145f20 msgr2=0x7f92801e33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9284a00640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9280145f20 0x7f92801e33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9284a00640 1 -- 172.21.15.62:0/2206307753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9280143e30 con 0x7f92800b9d90 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9284a00640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92800b9d90 0x7f9280147af0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f92700011f0 tx=0x7f927000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9275ffb640 1 -- 172.21.15.62:0/2206307753 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9270018070 con 0x7f92800b9d90 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9275ffb640 1 -- 172.21.15.62:0/2206307753 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92700074e0 con 0x7f92800b9d90 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9275ffb640 1 -- 172.21.15.62:0/2206307753 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9270004a10 con 0x7f92800b9d90 2024-04-18T10:06:28.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 -- 172.21.15.62:0/2206307753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9280144110 con 0x7f92800b9d90 2024-04-18T10:06:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.358+0000 7f9287465640 1 -- 172.21.15.62:0/2206307753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9280143060 con 0x7f92800b9d90 2024-04-18T10:06:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.362+0000 7f9287465640 1 -- 172.21.15.62:0/2206307753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9280068b90 con 0x7f92800b9d90 2024-04-18T10:06:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.362+0000 7f9275ffb640 1 -- 172.21.15.62:0/2206307753 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9270004bb0 con 0x7f92800b9d90 2024-04-18T10:06:28.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.362+0000 7f9275ffb640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9264068380 0x7f926406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.362+0000 7f9275ffb640 1 -- 172.21.15.62:0/2206307753 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f9270085c00 con 0x7f92800b9d90 2024-04-18T10:06:28.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.362+0000 7f9285201640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9264068380 0x7f926406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.362+0000 7f9285201640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9264068380 0x7f926406a840 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f927c0af000 tx=0x7f927c0a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:28.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.366+0000 7f9275ffb640 1 -- 172.21.15.62:0/2206307753 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f927008c080 con 0x7f92800b9d90 2024-04-18T10:06:28.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.522+0000 7f9287465640 1 -- 172.21.15.62:0/2206307753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f928009c210 con 0x7f92800b9d90 2024-04-18T10:06:28.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9275ffb640 1 -- 172.21.15.62:0/2206307753 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16593 (secure 0 0 0) 0x7f9270056250 con 0x7f92800b9d90 2024-04-18T10:06:28.523 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:28.524 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":21,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:00.219376+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:05:56.782649+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-18T10:05:57.541623+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:28.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9287465640 1 -- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9264068380 msgr2=0x7f926406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9287465640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9264068380 0x7f926406a840 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f927c0af000 tx=0x7f927c0a7070 comp rx=0 tx=0).stop 2024-04-18T10:06:28.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9287465640 1 -- 172.21.15.62:0/2206307753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92800b9d90 msgr2=0x7f9280147af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9287465640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92800b9d90 0x7f9280147af0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f92700011f0 tx=0x7f927000c9b0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9287465640 1 -- 172.21.15.62:0/2206307753 shutdown_connections 2024-04-18T10:06:28.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9287465640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9264068380 0x7f926406a840 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9287465640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9280145f20 0x7f92801e33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9287465640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f92800b9d90 0x7f9280147af0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9287465640 1 --2- 172.21.15.62:0/2206307753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f928009be30 0x7f9280142b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9287465640 1 -- 172.21.15.62:0/2206307753 >> 172.21.15.62:0/2206307753 conn(0x7f928006c0d0 msgr2=0x7f928009df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:28.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9287465640 1 -- 172.21.15.62:0/2206307753 shutdown_connections 2024-04-18T10:06:28.527 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.526+0000 7f9287465640 1 -- 172.21.15.62:0/2206307753 wait complete. 2024-04-18T10:06:28.542 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-18T10:06:28.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765fdf0640 1 Processor -- start 2024-04-18T10:06:28.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765fdf0640 1 -- start start 2024-04-18T10:06:28.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765fdf0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f765809b620 0x7f765809ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765fdf0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f765809c4d0 0x7f76581492b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765fdf0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f765809d430 con 0x7f76581497f0 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765fdf0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f765806eb30 con 0x7f765809b620 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765fdf0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f765809dea0 con 0x7f765809c4d0 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765e38d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f76581497f0 0x7f765806d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35378/0 (socket says 172.21.15.62:35378) 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765e38d640 1 -- 172.21.15.62:0/2788429877 learned_addr learned my addr 172.21.15.62:0/2788429877 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765db8c640 1 --2- 172.21.15.62:0/2788429877 >> v2:172.21.15.62:3300/0 conn(0x7f765809b620 0x7f765809ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765d38b640 1 --2- 172.21.15.62:0/2788429877 >> v2:172.21.15.145:3300/0 conn(0x7f765809c4d0 0x7f76581492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765cb8a640 1 -- 172.21.15.62:0/2788429877 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 703531740 0 0) 0x7f765809d430 con 0x7f76581497f0 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765cb8a640 1 -- 172.21.15.62:0/2788429877 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7644003530 con 0x7f76581497f0 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765db8c640 1 -- 172.21.15.62:0/2788429877 >> v2:172.21.15.145:3300/0 conn(0x7f765809c4d0 msgr2=0x7f76581492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765db8c640 1 --2- 172.21.15.62:0/2788429877 >> v2:172.21.15.145:3300/0 conn(0x7f765809c4d0 0x7f76581492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765db8c640 1 -- 172.21.15.62:0/2788429877 >> v1:172.21.15.62:6789/0 conn(0x7f76581497f0 legacy=0x7f765806d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.654+0000 7f765db8c640 1 -- 172.21.15.62:0/2788429877 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765809d6a0 con 0x7f765809b620 2024-04-18T10:06:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765db8c640 1 --2- 172.21.15.62:0/2788429877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765809b620 0x7f765809ba00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f765806e4d0 tx=0x7f765401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41eebbfa3968ded server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765cb8a640 1 -- 172.21.15.62:0/2788429877 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76540b0070 con 0x7f765809b620 2024-04-18T10:06:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765cb8a640 1 -- 172.21.15.62:0/2788429877 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76540a4e50 con 0x7f765809b620 2024-04-18T10:06:28.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765cb8a640 1 -- 172.21.15.62:0/2788429877 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76540ab820 con 0x7f765809b620 2024-04-18T10:06:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765fdf0640 1 -- 172.21.15.62:0/2788429877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765809b620 msgr2=0x7f765809ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765fdf0640 1 --2- 172.21.15.62:0/2788429877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765809b620 0x7f765809ba00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f765806e4d0 tx=0x7f765401c3c0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765fdf0640 1 -- 172.21.15.62:0/2788429877 shutdown_connections 2024-04-18T10:06:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765fdf0640 1 --2- 172.21.15.62:0/2788429877 >> v2:172.21.15.145:3300/0 conn(0x7f765809c4d0 0x7f76581492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765fdf0640 1 --2- 172.21.15.62:0/2788429877 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765809b620 0x7f765809ba00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765fdf0640 1 -- 172.21.15.62:0/2788429877 >> 172.21.15.62:0/2788429877 conn(0x7f76580b9420 msgr2=0x7f76580b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765fdf0640 1 -- 172.21.15.62:0/2788429877 shutdown_connections 2024-04-18T10:06:28.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765fdf0640 1 -- 172.21.15.62:0/2788429877 wait complete. 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765fdf0640 1 Processor -- start 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.658+0000 7f765fdf0640 1 -- start start 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765fdf0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f765809b620 0x7f76581e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765fdf0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f765809c4d0 0x7f76581e45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765fdf0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76581497f0 0x7f765809a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765fdf0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7658152c50 con 0x7f76581497f0 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765fdf0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7658152ad0 con 0x7f765809c4d0 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765fdf0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7658152dd0 con 0x7f765809b620 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765e38d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76581497f0 0x7f765809a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765e38d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76581497f0 0x7f765809a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37900/0 (socket says 172.21.15.62:37900) 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765e38d640 1 -- 172.21.15.62:0/2255646737 learned_addr learned my addr 172.21.15.62:0/2255646737 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765d38b640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f765809c4d0 0x7f76581e45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765e38d640 1 -- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f765809b620 msgr2=0x7f76581e4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765e38d640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f765809b620 0x7f76581e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765e38d640 1 -- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f765809c4d0 msgr2=0x7f76581e45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765e38d640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f765809c4d0 0x7f76581e45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765e38d640 1 -- 172.21.15.62:0/2255646737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7658148fe0 con 0x7f76581497f0 2024-04-18T10:06:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765e38d640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76581497f0 0x7f765809a4a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f765000ca70 tx=0x7f765000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f764e7fc640 1 -- 172.21.15.62:0/2255646737 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650017070 con 0x7f76581497f0 2024-04-18T10:06:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f764e7fc640 1 -- 172.21.15.62:0/2255646737 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f765000abd0 con 0x7f76581497f0 2024-04-18T10:06:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765fdf0640 1 -- 172.21.15.62:0/2255646737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76581e3b70 con 0x7f76581497f0 2024-04-18T10:06:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f764e7fc640 1 -- 172.21.15.62:0/2255646737 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650012400 con 0x7f76581497f0 2024-04-18T10:06:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765fdf0640 1 -- 172.21.15.62:0/2255646737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7658148bb0 con 0x7f76581497f0 2024-04-18T10:06:28.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765fdf0640 1 -- 172.21.15.62:0/2255646737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7658068b20 con 0x7f76581497f0 2024-04-18T10:06:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f764e7fc640 1 -- 172.21.15.62:0/2255646737 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f765001a070 con 0x7f76581497f0 2024-04-18T10:06:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f764e7fc640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f763c0683b0 0x7f763c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f764e7fc640 1 -- 172.21.15.62:0/2255646737 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7650086550 con 0x7f76581497f0 2024-04-18T10:06:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.662+0000 7f765db8c640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f763c0683b0 0x7f763c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:28.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.666+0000 7f765db8c640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f763c0683b0 0x7f763c06a870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f76540ad000 tx=0x7f7654004740 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:28.666 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.666+0000 7f764e7fc640 1 -- 172.21.15.62:0/2255646737 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7650056550 con 0x7f76581497f0 2024-04-18T10:06:28.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.858+0000 7f765fdf0640 1 -- 172.21.15.62:0/2255646737 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f765806e2b0 con 0x7f76581497f0 2024-04-18T10:06:28.857 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-18T10:06:28.858+0000 7f0d8301a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-18T10:06:28.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f764e7fc640 1 -- 172.21.15.62:0/2255646737 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7f765005a850 con 0x7f76581497f0 2024-04-18T10:06:28.859 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-18T10:06:28.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f765fdf0640 1 -- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f763c0683b0 msgr2=0x7f763c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f765fdf0640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f763c0683b0 0x7f763c06a870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f76540ad000 tx=0x7f7654004740 comp rx=0 tx=0).stop 2024-04-18T10:06:28.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f765fdf0640 1 -- 172.21.15.62:0/2255646737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76581497f0 msgr2=0x7f765809a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:28.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f765fdf0640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76581497f0 0x7f765809a4a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f765000ca70 tx=0x7f765000cf30 comp rx=0 tx=0).stop 2024-04-18T10:06:28.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f765fdf0640 1 -- 172.21.15.62:0/2255646737 shutdown_connections 2024-04-18T10:06:28.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f765fdf0640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f763c0683b0 0x7f763c06a870 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f765fdf0640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76581497f0 0x7f765809a4a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f765fdf0640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f765809c4d0 0x7f76581e45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.863 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f765fdf0640 1 --2- 172.21.15.62:0/2255646737 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f765809b620 0x7f76581e4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:28.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f765fdf0640 1 -- 172.21.15.62:0/2255646737 >> 172.21.15.62:0/2255646737 conn(0x7f76580b9420 msgr2=0x7f76580b7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:28.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f765fdf0640 1 -- 172.21.15.62:0/2255646737 shutdown_connections 2024-04-18T10:06:28.864 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:28.862+0000 7f765fdf0640 1 -- 172.21.15.62:0/2255646737 wait complete. 2024-04-18T10:06:28.864 INFO:tasks.ceph.mds.a.smithi062.stderr: -1159> 2024-04-18T10:06:02.278+0000 7f66aea66640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-18T10:06:28.878 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-18T10:06:28.879 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-18T10:06:28.898 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-04-18T10:06:28.899 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-18T10:06:29.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f55620ba640 1 Processor -- start 2024-04-18T10:06:29.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f55620ba640 1 -- start start 2024-04-18T10:06:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f55620ba640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f555c0ae4e0 0x7f555c13e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f55620ba640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f555c0b0580 con 0x7f555c0b01a0 2024-04-18T10:06:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f55620ba640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f555c0b59d0 con 0x7f555c0737c0 2024-04-18T10:06:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f55620ba640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f555c0ae970 con 0x7f555c0ae4e0 2024-04-18T10:06:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555b7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f555c0b01a0 0x7f555c06d2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35384/0 (socket says 172.21.15.62:35384) 2024-04-18T10:06:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555b7fe640 1 -- 172.21.15.62:0/3777427593 learned_addr learned my addr 172.21.15.62:0/3777427593 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 834310889 0 0) 0x7f555c0b0580 con 0x7f555c0b01a0 2024-04-18T10:06:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5544003530 con 0x7f555c0b01a0 2024-04-18T10:06:29.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555affd640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f555c0ae4e0 0x7f555c13e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2128192386 0 0) 0x7f5544003530 con 0x7f555c0b01a0 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f555c0ae4e0 msgr2=0x7f555c13e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f555c0ae4e0 0x7f555c13e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 >> v1:172.21.15.145:6789/0 conn(0x7f555c0737c0 legacy=0x7f555c14f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555c0b53a0 con 0x7f555c0b01a0 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f554c001680 con 0x7f555c0b01a0 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 >> v1:172.21.15.62:6789/0 conn(0x7f555c0b01a0 legacy=0x7f555c06d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55440039f0 0x7f5544003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5544004810 0x7f5544004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5544005190 0x7f554400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f555c0ae970 con 0x7f5544005190 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f554400c020 con 0x7f55440039f0 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f554400c220 con 0x7f5544004810 2024-04-18T10:06:29.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.006+0000 7f555bfff640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5544005190 0x7f554400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555affd640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5544004810 0x7f5544004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555b7fe640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55440039f0 0x7f5544003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555bfff640 1 -- 172.21.15.62:0/3777427593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5544004810 msgr2=0x7f5544004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555bfff640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5544004810 0x7f5544004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555bfff640 1 -- 172.21.15.62:0/3777427593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55440039f0 msgr2=0x7f5544003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555bfff640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55440039f0 0x7f5544003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555bfff640 1 -- 172.21.15.62:0/3777427593 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5544003530 con 0x7f5544005190 2024-04-18T10:06:29.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555b7fe640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55440039f0 0x7f5544003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:29.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555bfff640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5544005190 0x7f554400b9d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5554001e70 tx=0x7f555401c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1479cdd63fb7390 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:29.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5554001a90 con 0x7f5544005190 2024-04-18T10:06:29.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5554003b80 con 0x7f5544005190 2024-04-18T10:06:29.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555a7fc640 1 -- 172.21.15.62:0/3777427593 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c06e470 con 0x7f5544005190 2024-04-18T10:06:29.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f55620ba640 1 -- 172.21.15.62:0/3777427593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5544005190 msgr2=0x7f554400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:29.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f55620ba640 1 --2- 172.21.15.62:0/3777427593 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5544005190 0x7f554400b9d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5554001e70 tx=0x7f555401c780 comp rx=0 tx=0).stop 2024-04-18T10:06:29.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f555bfff640 1 -- 172.21.15.62:0/3777427593 reap_dead start 2024-04-18T10:06:29.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f55620ba640 1 -- 172.21.15.62:0/3777427593 shutdown_connections 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f55620ba640 1 -- 172.21.15.62:0/3777427593 >> 172.21.15.62:0/3777427593 conn(0x7f555c0b9690 msgr2=0x7f555c0b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f55620ba640 1 -- 172.21.15.62:0/3777427593 shutdown_connections 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f55620ba640 1 -- 172.21.15.62:0/3777427593 wait complete. 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f55620ba640 1 Processor -- start 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.010+0000 7f55620ba640 1 -- start start 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f55620ba640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f555c136c70 0x7f555c1de180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f55620ba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f555c1de6c0 0x7f555c1e16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f55620ba640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f555c1e1c30 0x7f555c1e20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f55620ba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f555c150ee0 con 0x7f555c1de6c0 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f55620ba640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f555c150d60 con 0x7f555c136c70 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f55620ba640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f555c151060 con 0x7f555c1e1c30 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f555affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f555c1de6c0 0x7f555c1e16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f555affd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f555c1de6c0 0x7f555c1e16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37924/0 (socket says 172.21.15.62:37924) 2024-04-18T10:06:29.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f555affd640 1 -- 172.21.15.62:0/2454775698 learned_addr learned my addr 172.21.15.62:0/2454775698 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f555b7fe640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f555c136c70 0x7f555c1de180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f555bfff640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f555c1e1c30 0x7f555c1e20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f555affd640 1 -- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f555c1e1c30 msgr2=0x7f555c1e20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f555affd640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f555c1e1c30 0x7f555c1e20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f555affd640 1 -- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f555c136c70 msgr2=0x7f555c1de180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f555affd640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f555c136c70 0x7f555c1de180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f555affd640 1 -- 172.21.15.62:0/2454775698 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f555c137050 con 0x7f555c1de6c0 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f555affd640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f555c1de6c0 0x7f555c1e16f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f555000f580 tx=0x7f555000fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f5543fff640 1 -- 172.21.15.62:0/2454775698 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550019020 con 0x7f555c1de6c0 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f5543fff640 1 -- 172.21.15.62:0/2454775698 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5550014870 con 0x7f555c1de6c0 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f55620ba640 1 -- 172.21.15.62:0/2454775698 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f555c138220 con 0x7f555c1de6c0 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f5543fff640 1 -- 172.21.15.62:0/2454775698 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5550014c40 con 0x7f555c1de6c0 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f55620ba640 1 -- 172.21.15.62:0/2454775698 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f555c1376d0 con 0x7f555c1de6c0 2024-04-18T10:06:29.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f55620ba640 1 -- 172.21.15.62:0/2454775698 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f555c068c00 con 0x7f555c1de6c0 2024-04-18T10:06:29.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f5543fff640 1 -- 172.21.15.62:0/2454775698 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f555001c020 con 0x7f555c1de6c0 2024-04-18T10:06:29.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f5543fff640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f55480683b0 0x7f554806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:29.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f5543fff640 1 -- 172.21.15.62:0/2454775698 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7f5550087190 con 0x7f555c1de6c0 2024-04-18T10:06:29.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.014+0000 7f555b7fe640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f55480683b0 0x7f554806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:29.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.018+0000 7f555b7fe640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f55480683b0 0x7f554806a870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f554c000c10 tx=0x7f554c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:29.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.018+0000 7f5543fff640 1 -- 172.21.15.62:0/2454775698 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5550023020 con 0x7f555c1de6c0 2024-04-18T10:06:29.216 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.218+0000 7f55620ba640 1 -- 172.21.15.62:0/2454775698 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f555c137b80 con 0x7f555c1de6c0 2024-04-18T10:06:29.867 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.866+0000 7f5543fff640 1 -- 172.21.15.62:0/2454775698 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7f5550057700 con 0x7f555c1de6c0 2024-04-18T10:06:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.870+0000 7f55620ba640 1 -- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f55480683b0 msgr2=0x7f554806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.870+0000 7f55620ba640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f55480683b0 0x7f554806a870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f554c000c10 tx=0x7f554c009040 comp rx=0 tx=0).stop 2024-04-18T10:06:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.870+0000 7f55620ba640 1 -- 172.21.15.62:0/2454775698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f555c1de6c0 msgr2=0x7f555c1e16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.870+0000 7f55620ba640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f555c1de6c0 0x7f555c1e16f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f555000f580 tx=0x7f555000fa40 comp rx=0 tx=0).stop 2024-04-18T10:06:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.870+0000 7f55620ba640 1 -- 172.21.15.62:0/2454775698 shutdown_connections 2024-04-18T10:06:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.870+0000 7f55620ba640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f55480683b0 0x7f554806a870 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.870+0000 7f55620ba640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f555c1e1c30 0x7f555c1e20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.870+0000 7f55620ba640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f555c1de6c0 0x7f555c1e16f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.870+0000 7f55620ba640 1 --2- 172.21.15.62:0/2454775698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f555c136c70 0x7f555c1de180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:29.871 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.870+0000 7f55620ba640 1 -- 172.21.15.62:0/2454775698 >> 172.21.15.62:0/2454775698 conn(0x7f555c0b9690 msgr2=0x7f555c13dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:29.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.870+0000 7f55620ba640 1 -- 172.21.15.62:0/2454775698 shutdown_connections 2024-04-18T10:06:29.872 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:29.870+0000 7f55620ba640 1 -- 172.21.15.62:0/2454775698 wait complete. 2024-04-18T10:06:29.896 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-18T10:06:30.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.074+0000 7f5669dce640 1 Processor -- start 2024-04-18T10:06:30.075 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.074+0000 7f5669dce640 1 -- start start 2024-04-18T10:06:30.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f5669dce640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f56640b3e20 0x7f56641461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:30.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f5669dce640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56640ba170 con 0x7f566406d920 2024-04-18T10:06:30.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f5669dce640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f566409cca0 con 0x7f56640b9d90 2024-04-18T10:06:30.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f5669dce640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f5664149930 con 0x7f56640b3e20 2024-04-18T10:06:30.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f5663fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f566406d920 0x7f5664150470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:35400/0 (socket says 172.21.15.62:35400) 2024-04-18T10:06:30.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f5663fff640 1 -- 172.21.15.62:0/3597581919 learned_addr learned my addr 172.21.15.62:0/3597581919 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:30.076 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f5662ffd640 1 --2- 172.21.15.62:0/3597581919 >> v2:172.21.15.145:3300/0 conn(0x7f56640b3e20 0x7f56641461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:30.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f56627fc640 1 -- 172.21.15.62:0/3597581919 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 782754971 0 0) 0x7f56640ba170 con 0x7f566406d920 2024-04-18T10:06:30.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f56627fc640 1 -- 172.21.15.62:0/3597581919 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f564c003530 con 0x7f566406d920 2024-04-18T10:06:30.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f56627fc640 1 -- 172.21.15.62:0/3597581919 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3702507947 0 0) 0x7f564c003530 con 0x7f566406d920 2024-04-18T10:06:30.077 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f5662ffd640 1 -- 172.21.15.62:0/3597581919 >> v1:172.21.15.145:6789/0 conn(0x7f56640b9d90 legacy=0x7f566406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:30.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f5662ffd640 1 -- 172.21.15.62:0/3597581919 >> v1:172.21.15.62:6789/0 conn(0x7f566406d920 legacy=0x7f5664150470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:30.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f5662ffd640 1 -- 172.21.15.62:0/3597581919 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5664149c10 con 0x7f56640b3e20 2024-04-18T10:06:30.078 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f5662ffd640 1 --2- 172.21.15.62:0/3597581919 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56640b3e20 0x7f56641461e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f56540097c0 tx=0x7f565401c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26c105b879b9bbe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:30.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f56627fc640 1 -- 172.21.15.62:0/3597581919 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56540b0070 con 0x7f56640b3e20 2024-04-18T10:06:30.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f56627fc640 1 -- 172.21.15.62:0/3597581919 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56540ab400 con 0x7f56640b3e20 2024-04-18T10:06:30.079 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.078+0000 7f56627fc640 1 -- 172.21.15.62:0/3597581919 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c001a00 con 0x7f56640b3e20 2024-04-18T10:06:30.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 -- 172.21.15.62:0/3597581919 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56640b3e20 msgr2=0x7f56641461e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:30.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 --2- 172.21.15.62:0/3597581919 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56640b3e20 0x7f56641461e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f56540097c0 tx=0x7f565401c330 comp rx=0 tx=0).stop 2024-04-18T10:06:30.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 -- 172.21.15.62:0/3597581919 shutdown_connections 2024-04-18T10:06:30.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 --2- 172.21.15.62:0/3597581919 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f56640b3e20 0x7f56641461e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:30.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 -- 172.21.15.62:0/3597581919 >> 172.21.15.62:0/3597581919 conn(0x7f566406c0d0 msgr2=0x7f56640b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:30.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 -- 172.21.15.62:0/3597581919 shutdown_connections 2024-04-18T10:06:30.080 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 -- 172.21.15.62:0/3597581919 wait complete. 2024-04-18T10:06:30.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 Processor -- start 2024-04-18T10:06:30.081 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 -- start start 2024-04-18T10:06:30.082 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f566406d920 0x7f566413fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56640b3e20 0x7f56641e2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56640b9d90 0x7f5664098580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f56641549d0 con 0x7f56640b9d90 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f566409c020 con 0x7f566406d920 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f5664154b50 con 0x7f56640b3e20 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5663fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56640b9d90 0x7f5664098580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5663fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56640b9d90 0x7f5664098580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:37928/0 (socket says 172.21.15.62:37928) 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5663fff640 1 -- 172.21.15.62:0/3760885169 learned_addr learned my addr 172.21.15.62:0/3760885169 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5662ffd640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56640b3e20 0x7f56641e2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f56637fe640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f566406d920 0x7f566413fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5663fff640 1 -- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56640b3e20 msgr2=0x7f56641e2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5663fff640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56640b3e20 0x7f56641e2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:30.083 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5663fff640 1 -- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f566406d920 msgr2=0x7f566413fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:30.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5663fff640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f566406d920 0x7f566413fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:30.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5663fff640 1 -- 172.21.15.62:0/3760885169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5664147300 con 0x7f56640b9d90 2024-04-18T10:06:30.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5663fff640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56640b9d90 0x7f5664098580 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f565c001570 tx=0x7f565c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:30.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f564bfff640 1 -- 172.21.15.62:0/3760885169 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c017070 con 0x7f56640b9d90 2024-04-18T10:06:30.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f564bfff640 1 -- 172.21.15.62:0/3760885169 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f565c00ad60 con 0x7f56640b9d90 2024-04-18T10:06:30.084 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 -- 172.21.15.62:0/3760885169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56640b0ca0 con 0x7f56640b9d90 2024-04-18T10:06:30.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f564bfff640 1 -- 172.21.15.62:0/3760885169 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c012640 con 0x7f56640b9d90 2024-04-18T10:06:30.085 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.082+0000 7f5669dce640 1 -- 172.21.15.62:0/3760885169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56640b09c0 con 0x7f56640b9d90 2024-04-18T10:06:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.086+0000 7f5669dce640 1 -- 172.21.15.62:0/3760885169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5664068ca0 con 0x7f56640b9d90 2024-04-18T10:06:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.086+0000 7f564bfff640 1 -- 172.21.15.62:0/3760885169 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f565c01a070 con 0x7f56640b9d90 2024-04-18T10:06:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.086+0000 7f564bfff640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56500683b0 0x7f565006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.086+0000 7f564bfff640 1 -- 172.21.15.62:0/3760885169 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5795+0+0 (secure 0 0 0) 0x7f565c086a60 con 0x7f56640b9d90 2024-04-18T10:06:30.086 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.086+0000 7f56637fe640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56500683b0 0x7f565006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:30.087 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.086+0000 7f56637fe640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56500683b0 0x7f565006a870 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f566413c780 tx=0x7f5658009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:30.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.090+0000 7f564bfff640 1 -- 172.21.15.62:0/3760885169 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f565c057050 con 0x7f56640b9d90 2024-04-18T10:06:30.252 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.254+0000 7f5669dce640 1 -- 172.21.15.62:0/3760885169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5664144c40 con 0x7f56640b9d90 2024-04-18T10:06:30.870 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.870+0000 7f564bfff640 1 -- 172.21.15.62:0/3760885169 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v23) ==== 181+0+0 (secure 0 0 0) 0x7f565c056950 con 0x7f56640b9d90 2024-04-18T10:06:30.870 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' removed 2024-04-18T10:06:30.875 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.878+0000 7f5669dce640 1 -- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56500683b0 msgr2=0x7f565006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:30.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.878+0000 7f5669dce640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56500683b0 0x7f565006a870 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f566413c780 tx=0x7f5658009040 comp rx=0 tx=0).stop 2024-04-18T10:06:30.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.878+0000 7f5669dce640 1 -- 172.21.15.62:0/3760885169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56640b9d90 msgr2=0x7f5664098580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:30.876 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.878+0000 7f5669dce640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56640b9d90 0x7f5664098580 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f565c001570 tx=0x7f565c00a430 comp rx=0 tx=0).stop 2024-04-18T10:06:30.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.878+0000 7f5669dce640 1 -- 172.21.15.62:0/3760885169 shutdown_connections 2024-04-18T10:06:30.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.878+0000 7f5669dce640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f56500683b0 0x7f565006a870 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:30.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.878+0000 7f5669dce640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56640b9d90 0x7f5664098580 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:30.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.878+0000 7f5669dce640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f56640b3e20 0x7f56641e2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:30.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.878+0000 7f5669dce640 1 --2- 172.21.15.62:0/3760885169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f566406d920 0x7f566413fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:30.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.878+0000 7f5669dce640 1 -- 172.21.15.62:0/3760885169 >> 172.21.15.62:0/3760885169 conn(0x7f566406c0d0 msgr2=0x7f56640aee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:30.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.878+0000 7f5669dce640 1 -- 172.21.15.62:0/3760885169 shutdown_connections 2024-04-18T10:06:30.877 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:30.878+0000 7f5669dce640 1 -- 172.21.15.62:0/3760885169 wait complete. 2024-04-18T10:06:30.898 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-18T10:06:31.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.030+0000 7f751ee59640 1 Processor -- start 2024-04-18T10:06:31.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751ee59640 1 -- start start 2024-04-18T10:06:31.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751ee59640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f75180b9d90 0x7f751806d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:31.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751ee59640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75180b3e20 0x7f751806d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:31.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751ee59640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f751806dc90 0x7f75181451d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:31.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751ee59640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f75180ba170 con 0x7f75180b9d90 2024-04-18T10:06:31.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751ee59640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f751809cca0 con 0x7f751806dc90 2024-04-18T10:06:31.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751ee59640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7518149930 con 0x7f75180b3e20 2024-04-18T10:06:31.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751cbf5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f75180b9d90 0x7f751806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:31.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751cbf5640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f75180b9d90 0x7f751806d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40824/0 (socket says 172.21.15.62:40824) 2024-04-18T10:06:31.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751cbf5640 1 -- 172.21.15.62:0/2075535090 learned_addr learned my addr 172.21.15.62:0/2075535090 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:31.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f7517fff640 1 --2- 172.21.15.62:0/2075535090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75180b3e20 0x7f751806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:31.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751cbf5640 1 -- 172.21.15.62:0/2075535090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75180b3e20 msgr2=0x7f751806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:31.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751d3f6640 1 --2- 172.21.15.62:0/2075535090 >> v2:172.21.15.145:3300/0 conn(0x7f751806dc90 0x7f75181451d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:31.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751cbf5640 1 --2- 172.21.15.62:0/2075535090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75180b3e20 0x7f751806d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:31.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751cbf5640 1 -- 172.21.15.62:0/2075535090 >> v2:172.21.15.145:3300/0 conn(0x7f751806dc90 msgr2=0x7f75181451d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:31.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751cbf5640 1 --2- 172.21.15.62:0/2075535090 >> v2:172.21.15.145:3300/0 conn(0x7f751806dc90 0x7f75181451d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:31.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751cbf5640 1 -- 172.21.15.62:0/2075535090 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7518149c10 con 0x7f75180b9d90 2024-04-18T10:06:31.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751d3f6640 1 --2- 172.21.15.62:0/2075535090 >> v2:172.21.15.145:3300/0 conn(0x7f751806dc90 0x7f75181451d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:31.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f751cbf5640 1 --2- 172.21.15.62:0/2075535090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180b9d90 0x7f751806d210 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f75180b3ae0 tx=0x7f750c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d886eddfa2c582d server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:31.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f75177fe640 1 -- 172.21.15.62:0/2075535090 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750c0b0070 con 0x7f75180b9d90 2024-04-18T10:06:31.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f75177fe640 1 -- 172.21.15.62:0/2075535090 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f750c0ab600 con 0x7f75180b9d90 2024-04-18T10:06:31.035 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.034+0000 7f75177fe640 1 -- 172.21.15.62:0/2075535090 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750c0ab9d0 con 0x7f75180b9d90 2024-04-18T10:06:31.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 -- 172.21.15.62:0/2075535090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180b9d90 msgr2=0x7f751806d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:31.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 --2- 172.21.15.62:0/2075535090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180b9d90 0x7f751806d210 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f75180b3ae0 tx=0x7f750c01c470 comp rx=0 tx=0).stop 2024-04-18T10:06:31.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 -- 172.21.15.62:0/2075535090 shutdown_connections 2024-04-18T10:06:31.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 --2- 172.21.15.62:0/2075535090 >> v2:172.21.15.145:3300/0 conn(0x7f751806dc90 0x7f75181451d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:31.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 --2- 172.21.15.62:0/2075535090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75180b3e20 0x7f751806d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:31.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 --2- 172.21.15.62:0/2075535090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180b9d90 0x7f751806d210 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:31.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 -- 172.21.15.62:0/2075535090 >> 172.21.15.62:0/2075535090 conn(0x7f751806c0d0 msgr2=0x7f75180b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:31.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 -- 172.21.15.62:0/2075535090 shutdown_connections 2024-04-18T10:06:31.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 -- 172.21.15.62:0/2075535090 wait complete. 2024-04-18T10:06:31.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 Processor -- start 2024-04-18T10:06:31.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 -- start start 2024-04-18T10:06:31.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f751806dc90 0x7f75181e18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:31.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75180b3e20 0x7f751813e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:31.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180b9d90 0x7f75180b1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:31.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7518146130 con 0x7f75180b9d90 2024-04-18T10:06:31.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f751806e0a0 con 0x7f751806dc90 2024-04-18T10:06:31.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f75181462b0 con 0x7f75180b3e20 2024-04-18T10:06:31.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751d3f6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180b9d90 0x7f75180b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:31.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751d3f6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180b9d90 0x7f75180b1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40826/0 (socket says 172.21.15.62:40826) 2024-04-18T10:06:31.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751d3f6640 1 -- 172.21.15.62:0/1826422106 learned_addr learned my addr 172.21.15.62:0/1826422106 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:31.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751cbf5640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f751806dc90 0x7f75181e18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:31.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f7517fff640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75180b3e20 0x7f751813e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:31.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751d3f6640 1 -- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75180b3e20 msgr2=0x7f751813e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:31.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751d3f6640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75180b3e20 0x7f751813e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:31.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751d3f6640 1 -- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f751806dc90 msgr2=0x7f75181e18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:31.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751d3f6640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f751806dc90 0x7f75181e18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:31.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751d3f6640 1 -- 172.21.15.62:0/1826422106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75181e23b0 con 0x7f75180b9d90 2024-04-18T10:06:31.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751cbf5640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f751806dc90 0x7f75181e18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:31.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751d3f6640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180b9d90 0x7f75180b1650 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f751000cef0 tx=0x7f751000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:31.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f75157fa640 1 -- 172.21.15.62:0/1826422106 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751001a070 con 0x7f75180b9d90 2024-04-18T10:06:31.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f75157fa640 1 -- 172.21.15.62:0/1826422106 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f751000d840 con 0x7f75180b9d90 2024-04-18T10:06:31.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 -- 172.21.15.62:0/1826422106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7518069570 con 0x7f75180b9d90 2024-04-18T10:06:31.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f75157fa640 1 -- 172.21.15.62:0/1826422106 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751000dc30 con 0x7f75180b9d90 2024-04-18T10:06:31.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.038+0000 7f751ee59640 1 -- 172.21.15.62:0/1826422106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f751813ebc0 con 0x7f75180b9d90 2024-04-18T10:06:31.042 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.042+0000 7f751ee59640 1 -- 172.21.15.62:0/1826422106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7518068ca0 con 0x7f75180b9d90 2024-04-18T10:06:31.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.042+0000 7f75157fa640 1 -- 172.21.15.62:0/1826422106 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f751001d070 con 0x7f75180b9d90 2024-04-18T10:06:31.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.042+0000 7f75157fa640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f74fc068380 0x7f74fc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:31.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.042+0000 7f75157fa640 1 -- 172.21.15.62:0/1826422106 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5357+0+0 (secure 0 0 0) 0x7f7510086fd0 con 0x7f75180b9d90 2024-04-18T10:06:31.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.042+0000 7f751cbf5640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f74fc068380 0x7f74fc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:31.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.042+0000 7f751cbf5640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f74fc068380 0x7f74fc06a840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f750c0ad000 tx=0x7f750c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:31.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.046+0000 7f75157fa640 1 -- 172.21.15.62:0/1826422106 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7510057780 con 0x7f75180b9d90 2024-04-18T10:06:31.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.210+0000 7f751ee59640 1 -- 172.21.15.62:0/1826422106 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f75180af360 con 0x7f75180b9d90 2024-04-18T10:06:31.878 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.878+0000 7f75157fa640 1 -- 172.21.15.62:0/1826422106 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v24) ==== 169+0+0 (secure 0 0 0) 0x7f7510057080 con 0x7f75180b9d90 2024-04-18T10:06:31.878 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' removed 2024-04-18T10:06:31.883 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.882+0000 7f751ee59640 1 -- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f74fc068380 msgr2=0x7f74fc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:31.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.882+0000 7f751ee59640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f74fc068380 0x7f74fc06a840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f750c0ad000 tx=0x7f750c008040 comp rx=0 tx=0).stop 2024-04-18T10:06:31.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.882+0000 7f751ee59640 1 -- 172.21.15.62:0/1826422106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180b9d90 msgr2=0x7f75180b1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:31.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.882+0000 7f751ee59640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180b9d90 0x7f75180b1650 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f751000cef0 tx=0x7f751000a710 comp rx=0 tx=0).stop 2024-04-18T10:06:31.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.886+0000 7f751ee59640 1 -- 172.21.15.62:0/1826422106 shutdown_connections 2024-04-18T10:06:31.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.886+0000 7f751ee59640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f74fc068380 0x7f74fc06a840 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:31.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.886+0000 7f751ee59640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f75180b9d90 0x7f75180b1650 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:31.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.886+0000 7f751ee59640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75180b3e20 0x7f751813e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:31.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.886+0000 7f751ee59640 1 --2- 172.21.15.62:0/1826422106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f751806dc90 0x7f75181e18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:31.884 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.886+0000 7f751ee59640 1 -- 172.21.15.62:0/1826422106 >> 172.21.15.62:0/1826422106 conn(0x7f751806c0d0 msgr2=0x7f751809d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:31.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.886+0000 7f751ee59640 1 -- 172.21.15.62:0/1826422106 shutdown_connections 2024-04-18T10:06:31.885 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:31.886+0000 7f751ee59640 1 -- 172.21.15.62:0/1826422106 wait complete. 2024-04-18T10:06:31.907 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-18T10:06:31.907 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-18T10:06:31.907 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-18T10:06:31.908 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:06:31.908 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-18T10:06:31.908 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-18T10:06:31.908 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-18T10:06:31.908 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:06:31.918 INFO:tasks.ceph.mds.a:Stopped 2024-04-18T10:06:31.918 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-18T10:06:31.931 INFO:tasks.ceph.mds.b:Stopped 2024-04-18T10:06:31.931 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-18T10:06:31.961 INFO:tasks.ceph.mds.a:Started 2024-04-18T10:06:31.961 DEBUG:teuthology.parallel:result is None 2024-04-18T10:06:31.978 INFO:tasks.ceph.mds.b:Started 2024-04-18T10:06:31.978 DEBUG:teuthology.parallel:result is None 2024-04-18T10:06:31.978 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-18T10:06:32.038 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-04-18T10:06:32.053 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-04-18T10:06:32.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.106+0000 7f9bd8352640 1 Processor -- start 2024-04-18T10:06:32.107 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.106+0000 7f9bd8352640 1 -- start start 2024-04-18T10:06:32.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd8352640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9bd0067a70 0x7f9bd013faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:32.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd8352640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bd00aeda0 0x7f9bd01498b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:32.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd8352640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bd0067e50 con 0x7f9bd013ffe0 2024-04-18T10:06:32.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd8352640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f9bd0146b60 con 0x7f9bd0067a70 2024-04-18T10:06:32.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd8352640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd0149140 con 0x7f9bd00aeda0 2024-04-18T10:06:32.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd60ee640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9bd0067a70 0x7f9bd013faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:32.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd68ef640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9bd013ffe0 0x7f9bd006bad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33830/0 (socket says 172.21.15.62:33830) 2024-04-18T10:06:32.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd68ef640 1 -- 172.21.15.62:0/2296293286 learned_addr learned my addr 172.21.15.62:0/2296293286 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:32.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd58ed640 1 --2- 172.21.15.62:0/2296293286 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bd00aeda0 0x7f9bd01498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:32.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd60ee640 1 -- 172.21.15.62:0/2296293286 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bd00aeda0 msgr2=0x7f9bd01498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:32.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd60ee640 1 --2- 172.21.15.62:0/2296293286 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bd00aeda0 0x7f9bd01498b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:32.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd60ee640 1 -- 172.21.15.62:0/2296293286 >> v1:172.21.15.62:6789/0 conn(0x7f9bd013ffe0 legacy=0x7f9bd006bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:32.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd50ec640 1 -- 172.21.15.62:0/2296293286 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 649609629 0 0) 0x7f9bd0067e50 con 0x7f9bd013ffe0 2024-04-18T10:06:32.109 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd60ee640 1 -- 172.21.15.62:0/2296293286 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd00b0040 con 0x7f9bd0067a70 2024-04-18T10:06:32.110 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd60ee640 1 --2- 172.21.15.62:0/2296293286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd0067a70 0x7f9bd013faa0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0097c0 tx=0x7f9bcc01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738a0818880c9f57 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:32.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd50ec640 1 -- 172.21.15.62:0/2296293286 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc0ab610 con 0x7f9bd0067a70 2024-04-18T10:06:32.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd50ec640 1 -- 172.21.15.62:0/2296293286 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bcc0b0070 con 0x7f9bd0067a70 2024-04-18T10:06:32.113 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.110+0000 7f9bd50ec640 1 -- 172.21.15.62:0/2296293286 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc0ab9d0 con 0x7f9bd0067a70 2024-04-18T10:06:32.116 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 -- 172.21.15.62:0/2296293286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd0067a70 msgr2=0x7f9bd013faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 --2- 172.21.15.62:0/2296293286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd0067a70 0x7f9bd013faa0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0097c0 tx=0x7f9bcc01c410 comp rx=0 tx=0).stop 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 -- 172.21.15.62:0/2296293286 shutdown_connections 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 --2- 172.21.15.62:0/2296293286 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bd00aeda0 0x7f9bd01498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 --2- 172.21.15.62:0/2296293286 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd0067a70 0x7f9bd013faa0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 -- 172.21.15.62:0/2296293286 >> 172.21.15.62:0/2296293286 conn(0x7f9bd00b9c80 msgr2=0x7f9bd00ba090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 -- 172.21.15.62:0/2296293286 shutdown_connections 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 -- 172.21.15.62:0/2296293286 wait complete. 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 Processor -- start 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 -- start start 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bd0067a70 0x7f9bd00b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd00aeda0 0x7f9bd01e2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bd013ffe0 0x7f9bd009c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd0153670 con 0x7f9bd00aeda0 2024-04-18T10:06:32.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd01534f0 con 0x7f9bd013ffe0 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd01537f0 con 0x7f9bd0067a70 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd58ed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd00aeda0 0x7f9bd01e2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd58ed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd00aeda0 0x7f9bd01e2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40866/0 (socket says 172.21.15.62:40866) 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd58ed640 1 -- 172.21.15.62:0/1907484645 learned_addr learned my addr 172.21.15.62:0/1907484645 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd60ee640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bd0067a70 0x7f9bd00b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd68ef640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bd013ffe0 0x7f9bd009c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd58ed640 1 -- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bd0067a70 msgr2=0x7f9bd00b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd58ed640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bd0067a70 0x7f9bd00b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd58ed640 1 -- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bd013ffe0 msgr2=0x7f9bd009c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd58ed640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bd013ffe0 0x7f9bd009c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd58ed640 1 -- 172.21.15.62:0/1907484645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd009da00 con 0x7f9bd00aeda0 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd68ef640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bd013ffe0 0x7f9bd009c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd58ed640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd00aeda0 0x7f9bd01e2640 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000d830 tx=0x7f9bc000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bc6ffd640 1 -- 172.21.15.62:0/1907484645 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc000b6c0 con 0x7f9bd00aeda0 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bc6ffd640 1 -- 172.21.15.62:0/1907484645 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc0019070 con 0x7f9bd00aeda0 2024-04-18T10:06:32.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 -- 172.21.15.62:0/1907484645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd013fa70 con 0x7f9bd00aeda0 2024-04-18T10:06:32.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bc6ffd640 1 -- 172.21.15.62:0/1907484645 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc000ba60 con 0x7f9bd00aeda0 2024-04-18T10:06:32.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 -- 172.21.15.62:0/1907484645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd01e2120 con 0x7f9bd00aeda0 2024-04-18T10:06:32.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.114+0000 7f9bd8352640 1 -- 172.21.15.62:0/1907484645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd006cd50 con 0x7f9bd00aeda0 2024-04-18T10:06:32.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.118+0000 7f9bc6ffd640 1 -- 172.21.15.62:0/1907484645 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9bc001e460 con 0x7f9bd00aeda0 2024-04-18T10:06:32.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.118+0000 7f9bc6ffd640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9bb4068380 0x7f9bb406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:32.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.118+0000 7f9bc6ffd640 1 -- 172.21.15.62:0/1907484645 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 4943+0+0 (secure 0 0 0) 0x7f9bc00876d0 con 0x7f9bd00aeda0 2024-04-18T10:06:32.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.118+0000 7f9bd60ee640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9bb4068380 0x7f9bb406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:32.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.118+0000 7f9bd60ee640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9bb4068380 0x7f9bb406a840 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0af000 tx=0x7f9bcc0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:32.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.122+0000 7f9bc6ffd640 1 -- 172.21.15.62:0/1907484645 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9bc0058100 con 0x7f9bd00aeda0 2024-04-18T10:06:32.294 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.294+0000 7f9bd8352640 1 -- 172.21.15.62:0/1907484645 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f9bd0068b20 con 0x7f9bd00aeda0 2024-04-18T10:06:32.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.902+0000 7f9bc6ffd640 1 -- 172.21.15.62:0/1907484645 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v26) ==== 97+0+0 (secure 0 0 0) 0x7f9bc0057a00 con 0x7f9bd00aeda0 2024-04-18T10:06:32.902 INFO:teuthology.orchestra.run.smithi062.stderr: removed all blocklist entries 2024-04-18T10:06:32.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.906+0000 7f9bd8352640 1 -- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9bb4068380 msgr2=0x7f9bb406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:32.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.906+0000 7f9bd8352640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9bb4068380 0x7f9bb406a840 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0af000 tx=0x7f9bcc0a7000 comp rx=0 tx=0).stop 2024-04-18T10:06:32.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.906+0000 7f9bd8352640 1 -- 172.21.15.62:0/1907484645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd00aeda0 msgr2=0x7f9bd01e2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:32.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.906+0000 7f9bd8352640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd00aeda0 0x7f9bd01e2640 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000d830 tx=0x7f9bc000dcf0 comp rx=0 tx=0).stop 2024-04-18T10:06:32.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.906+0000 7f9bd8352640 1 -- 172.21.15.62:0/1907484645 shutdown_connections 2024-04-18T10:06:32.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.906+0000 7f9bd8352640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9bb4068380 0x7f9bb406a840 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:32.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.906+0000 7f9bd8352640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bd013ffe0 0x7f9bd009c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:32.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.906+0000 7f9bd8352640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9bd00aeda0 0x7f9bd01e2640 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:32.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.906+0000 7f9bd8352640 1 --2- 172.21.15.62:0/1907484645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bd0067a70 0x7f9bd00b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:32.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.906+0000 7f9bd8352640 1 -- 172.21.15.62:0/1907484645 >> 172.21.15.62:0/1907484645 conn(0x7f9bd00b9c80 msgr2=0x7f9bd00b6d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:32.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.906+0000 7f9bd8352640 1 -- 172.21.15.62:0/1907484645 shutdown_connections 2024-04-18T10:06:32.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:32.906+0000 7f9bd8352640 1 -- 172.21.15.62:0/1907484645 wait complete. 2024-04-18T10:06:32.924 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-18T10:06:33.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829cee8640 1 Processor -- start 2024-04-18T10:06:33.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829cee8640 1 -- start start 2024-04-18T10:06:33.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829cee8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82980b6640 0x7f82981419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829cee8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82980b7850 con 0x7f8298141ee0 2024-04-18T10:06:33.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829cee8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82980b4c20 con 0x7f82980b6260 2024-04-18T10:06:33.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829cee8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8298148cb0 con 0x7f82980b6640 2024-04-18T10:06:33.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f8296d9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8298141ee0 0x7f82980b5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33832/0 (socket says 172.21.15.62:33832) 2024-04-18T10:06:33.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f8296d9d640 1 -- 172.21.15.62:0/3954088049 learned_addr learned my addr 172.21.15.62:0/3954088049 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:33.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f8295d9b640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82980b6640 0x7f82981419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3048066312 0 0) 0x7f82980b7850 con 0x7f8298141ee0 2024-04-18T10:06:33.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8280003530 con 0x7f8298141ee0 2024-04-18T10:06:33.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3518022774 0 0) 0x7f8280003530 con 0x7f8298141ee0 2024-04-18T10:06:33.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82980b6640 msgr2=0x7f82981419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:33.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82980b6640 0x7f82981419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 >> v1:172.21.15.145:6789/0 conn(0x7f82980b6260 legacy=0x7f8298148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8298148970 con 0x7f8298141ee0 2024-04-18T10:06:33.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f8290001a00 con 0x7f8298141ee0 2024-04-18T10:06:33.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 >> v1:172.21.15.62:6789/0 conn(0x7f8298141ee0 legacy=0x7f82980b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.047 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82800039f0 0x7f8280003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82980b6640 0x7f8280004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8280004d50 0x7f82800051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8298148cb0 con 0x7f82800039f0 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f828000bbb0 con 0x7f8280004d50 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f828000bdb0 con 0x7f82980b6640 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829659c640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82800039f0 0x7f8280003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f8296d9d640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8280004d50 0x7f82800051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f8295d9b640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82980b6640 0x7f8280004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829659c640 1 -- 172.21.15.62:0/3954088049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82980b6640 msgr2=0x7f8280004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829659c640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82980b6640 0x7f8280004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829659c640 1 -- 172.21.15.62:0/3954088049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8280004d50 msgr2=0x7f82800051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829659c640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8280004d50 0x7f82800051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f829659c640 1 -- 172.21.15.62:0/3954088049 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8280003530 con 0x7f82800039f0 2024-04-18T10:06:33.048 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.046+0000 7f8296d9d640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8280004d50 0x7f82800051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:33.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829659c640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82800039f0 0x7f8280003e30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8288001780 tx=0x7f828801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e273111257138ad server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:33.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82880013a0 con 0x7f82800039f0 2024-04-18T10:06:33.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82880052f0 con 0x7f82800039f0 2024-04-18T10:06:33.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829559a640 1 -- 172.21.15.62:0/3954088049 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82880a4e60 con 0x7f82800039f0 2024-04-18T10:06:33.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 -- 172.21.15.62:0/3954088049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82800039f0 msgr2=0x7f8280003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 --2- 172.21.15.62:0/3954088049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82800039f0 0x7f8280003e30 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8288001780 tx=0x7f828801c420 comp rx=0 tx=0).stop 2024-04-18T10:06:33.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f8296d9d640 1 -- 172.21.15.62:0/3954088049 reap_dead start 2024-04-18T10:06:33.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 -- 172.21.15.62:0/3954088049 shutdown_connections 2024-04-18T10:06:33.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 -- 172.21.15.62:0/3954088049 >> 172.21.15.62:0/3954088049 conn(0x7f82980bd5c0 msgr2=0x7f82980bd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:33.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 -- 172.21.15.62:0/3954088049 shutdown_connections 2024-04-18T10:06:33.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 -- 172.21.15.62:0/3954088049 wait complete. 2024-04-18T10:06:33.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 Processor -- start 2024-04-18T10:06:33.051 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 -- start start 2024-04-18T10:06:33.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f829814acd0 0x7f82981e0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82981e14c0 0x7f82981e3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82981e4440 0x7f82981e48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8298154240 con 0x7f82981e14c0 2024-04-18T10:06:33.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f82981540c0 con 0x7f82981e4440 2024-04-18T10:06:33.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.050+0000 7f829cee8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f82981543c0 con 0x7f829814acd0 2024-04-18T10:06:33.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8295d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82981e14c0 0x7f82981e3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8295d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82981e14c0 0x7f82981e3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40892/0 (socket says 172.21.15.62:40892) 2024-04-18T10:06:33.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8295d9b640 1 -- 172.21.15.62:0/2019744434 learned_addr learned my addr 172.21.15.62:0/2019744434 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:33.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f829659c640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f829814acd0 0x7f82981e0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8296d9d640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82981e4440 0x7f82981e48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8295d9b640 1 -- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f829814acd0 msgr2=0x7f82981e0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8295d9b640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f829814acd0 0x7f82981e0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8295d9b640 1 -- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82981e4440 msgr2=0x7f82981e48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8295d9b640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82981e4440 0x7f82981e48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8295d9b640 1 -- 172.21.15.62:0/2019744434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8298143ab0 con 0x7f82981e14c0 2024-04-18T10:06:33.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8295d9b640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82981e14c0 0x7f82981e3f00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f828c0108c0 tx=0x7f828c010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:33.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8286ffd640 1 -- 172.21.15.62:0/2019744434 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c00f040 con 0x7f82981e14c0 2024-04-18T10:06:33.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8286ffd640 1 -- 172.21.15.62:0/2019744434 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f828c00eb60 con 0x7f82981e14c0 2024-04-18T10:06:33.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f829cee8640 1 -- 172.21.15.62:0/2019744434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82981e0aa0 con 0x7f82981e14c0 2024-04-18T10:06:33.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8286ffd640 1 -- 172.21.15.62:0/2019744434 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f828c01c400 con 0x7f82981e14c0 2024-04-18T10:06:33.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f829cee8640 1 -- 172.21.15.62:0/2019744434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82981425e0 con 0x7f82981e14c0 2024-04-18T10:06:33.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f829cee8640 1 -- 172.21.15.62:0/2019744434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8298073bf0 con 0x7f82981e14c0 2024-04-18T10:06:33.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8286ffd640 1 -- 172.21.15.62:0/2019744434 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f828c01c5a0 con 0x7f82981e14c0 2024-04-18T10:06:33.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8286ffd640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8278068380 0x7f827806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f8286ffd640 1 -- 172.21.15.62:0/2019744434 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f828c086b90 con 0x7f82981e14c0 2024-04-18T10:06:33.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f829659c640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8278068380 0x7f827806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.054+0000 7f829659c640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8278068380 0x7f827806a840 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8288001750 tx=0x7f82880011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:33.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.058+0000 7f8286ffd640 1 -- 172.21.15.62:0/2019744434 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f828c018030 con 0x7f82981e14c0 2024-04-18T10:06:33.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.282+0000 7f829cee8640 1 -- 172.21.15.62:0/2019744434 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f8298069440 con 0x7f82981e14c0 2024-04-18T10:06:33.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.282+0000 7f8286ffd640 1 -- 172.21.15.62:0/2019744434 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+5207 (secure 0 0 0) 0x7f828c057530 con 0x7f82981e14c0 2024-04-18T10:06:33.282 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout: "auth_dump": [ 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.a", 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBN8CBmQwZLBBAAd+7JyRw7FcVA0o4nq9is7w==", 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.283 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mds.b", 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBN8CBmG07RBxAAfwMp6QSjdomEaKVUwxaQqA==", 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.0", 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBQ8CBmW55eCRAA800gMwDdNIxWAtzhDjuuMA==", 2024-04-18T10:06:33.284 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.1", 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBT8CBmj9V2DhAAonHttsgbVKgzZoht9NyVjA==", 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.285 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.2", 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBW8CBm6CJcFRAADOM4tR+oWmS24E7lZQ0i/Q==", 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.3", 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBZ8CBmt/YZHBAAniWnP+/mo8DmMHvG1jujlg==", 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.286 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.4", 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBf8CBmpsGDGBAAj9fJH1NSwQT+EjebyYw04g==", 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.287 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.5", 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBi8CBm5LD4HBAAtfN4fLfIAacGG87VNrDFaQ==", 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.6", 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBl8CBmP4t3JhAAls1hDJHR8ldfkyMg9isu1Q==", 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-04-18T10:06:33.288 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "osd.7", 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBo8CBm4iwTLRAAKiomOOGDcB7xyrS1aezjYw==", 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow profile osd", 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile osd", 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.0", 2024-04-18T10:06:33.289 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBN8CBmSFtIDRAAjl9lXubksjAuIJZdtH8x2Q==", 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow", 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow r", 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow rw", 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow rwx" 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.admin", 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBM8CBm3v6tJxAASnxL1hArq1ll8iWZTFj4bw==", 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-04-18T10:06:33.290 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": "allow *", 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow *", 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.bootstrap-mds", 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBu8CBmEFF7FBAAglgqzFhNMQ1kFHH9W063hQ==", 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile bootstrap-mds" 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.bootstrap-mgr", 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBu8CBm+2l7FBAAYRgbP5pShWZy4CsW1TGkdA==", 2024-04-18T10:06:33.291 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.bootstrap-osd", 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBu8CBmH4J7FBAAHQDOXbaTinQJMknYvxzFRQ==", 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile bootstrap-osd" 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.bootstrap-rbd", 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBu8CBm65d7FBAAqJmajMagbwJyv45BKdziXQ==", 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-18T10:06:33.292 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBu8CBmkq17FBAAnoAj9L0CL9AktlsSml4SqQ==", 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "client.bootstrap-rgw", 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBu8CBmz8N7FBAA5CXALIAUwe753II4pEvvsA==", 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.293 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.x", 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBM8CBmS9qDOxAAV3nS9EtJXwkY2e1xQlhsvQ==", 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: "entity": "mgr.y", 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "AQBM8CBmkBCfOBAACzvtC9y+pLGAftyUQAmg2Q==", 2024-04-18T10:06:33.294 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": { 2024-04-18T10:06:33.295 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": "allow *", 2024-04-18T10:06:33.295 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": "allow profile mgr", 2024-04-18T10:06:33.295 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": "allow *" 2024-04-18T10:06:33.295 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.295 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:33.295 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-04-18T10:06:33.295 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:06:33.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.286+0000 7f829cee8640 1 -- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8278068380 msgr2=0x7f827806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.286+0000 7f829cee8640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8278068380 0x7f827806a840 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8288001750 tx=0x7f82880011a0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.286+0000 7f829cee8640 1 -- 172.21.15.62:0/2019744434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82981e14c0 msgr2=0x7f82981e3f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.286+0000 7f829cee8640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82981e14c0 0x7f82981e3f00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f828c0108c0 tx=0x7f828c010d80 comp rx=0 tx=0).stop 2024-04-18T10:06:33.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.286+0000 7f829cee8640 1 -- 172.21.15.62:0/2019744434 shutdown_connections 2024-04-18T10:06:33.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.286+0000 7f829cee8640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8278068380 0x7f827806a840 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.286+0000 7f829cee8640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82981e4440 0x7f82981e48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.286+0000 7f829cee8640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82981e14c0 0x7f82981e3f00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.286+0000 7f829cee8640 1 --2- 172.21.15.62:0/2019744434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f829814acd0 0x7f82981e0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.286+0000 7f829cee8640 1 -- 172.21.15.62:0/2019744434 >> 172.21.15.62:0/2019744434 conn(0x7f82980bd5c0 msgr2=0x7f82980ba720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:33.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.286+0000 7f829cee8640 1 -- 172.21.15.62:0/2019744434 shutdown_connections 2024-04-18T10:06:33.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.286+0000 7f829cee8640 1 -- 172.21.15.62:0/2019744434 wait complete. 2024-04-18T10:06:33.301 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-18T10:06:33.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.425+0000 7fc27c0bc640 1 Processor -- start 2024-04-18T10:06:33.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.425+0000 7fc27c0bc640 1 -- start start 2024-04-18T10:06:33.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.425+0000 7fc27c0bc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc27409d2e0 0x7fc27406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.425+0000 7fc27c0bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2740ae6e0 0x7fc274073a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.425+0000 7fc27c0bc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc274144270 con 0x7fc274144a30 2024-04-18T10:06:33.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.425+0000 7fc27c0bc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fc274074110 con 0x7fc27409d2e0 2024-04-18T10:06:33.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.425+0000 7fc27c0bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc27406d920 con 0x7fc2740ae6e0 2024-04-18T10:06:33.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc279e58640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc27409d2e0 0x7fc27406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc279e58640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fc27409d2e0 0x7fc27406dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40908/0 (socket says 172.21.15.62:40908) 2024-04-18T10:06:33.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc279e58640 1 -- 172.21.15.62:0/1645799506 learned_addr learned my addr 172.21.15.62:0/1645799506 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:33.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc279657640 1 --2- 172.21.15.62:0/1645799506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2740ae6e0 0x7fc274073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc279e58640 1 -- 172.21.15.62:0/1645799506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2740ae6e0 msgr2=0x7fc274073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:33.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc279e58640 1 --2- 172.21.15.62:0/1645799506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2740ae6e0 0x7fc274073a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc279e58640 1 -- 172.21.15.62:0/1645799506 >> v1:172.21.15.145:6789/0 conn(0x7fc274144a30 legacy=0x7fc27406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:33.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc279e58640 1 -- 172.21.15.62:0/1645799506 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27409c4d0 con 0x7fc27409d2e0 2024-04-18T10:06:33.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc279e58640 1 --2- 172.21.15.62:0/1645799506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc27409d2e0 0x7fc27406dd00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc270009870 tx=0x7fc27001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48911df9bc4aa7a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:33.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc278e56640 1 -- 172.21.15.62:0/1645799506 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2700b0070 con 0x7fc27409d2e0 2024-04-18T10:06:33.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc278e56640 1 -- 172.21.15.62:0/1645799506 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2700ab600 con 0x7fc27409d2e0 2024-04-18T10:06:33.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc278e56640 1 -- 172.21.15.62:0/1645799506 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27413e7b0 con 0x7fc27409d2e0 2024-04-18T10:06:33.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc27c0bc640 1 -- 172.21.15.62:0/1645799506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc27409d2e0 msgr2=0x7fc27406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc27c0bc640 1 --2- 172.21.15.62:0/1645799506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc27409d2e0 0x7fc27406dd00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc270009870 tx=0x7fc27001c4d0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc27c0bc640 1 -- 172.21.15.62:0/1645799506 shutdown_connections 2024-04-18T10:06:33.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc27c0bc640 1 --2- 172.21.15.62:0/1645799506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2740ae6e0 0x7fc274073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc27c0bc640 1 --2- 172.21.15.62:0/1645799506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc27409d2e0 0x7fc27406dd00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc27c0bc640 1 -- 172.21.15.62:0/1645799506 >> 172.21.15.62:0/1645799506 conn(0x7fc2740b9420 msgr2=0x7fc2740b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:33.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc27c0bc640 1 -- 172.21.15.62:0/1645799506 shutdown_connections 2024-04-18T10:06:33.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.429+0000 7fc27c0bc640 1 -- 172.21.15.62:0/1645799506 wait complete. 2024-04-18T10:06:33.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc27c0bc640 1 Processor -- start 2024-04-18T10:06:33.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc27c0bc640 1 -- start start 2024-04-18T10:06:33.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc27c0bc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc27409d2e0 0x7fc274140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc27c0bc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2740ae6e0 0x7fc2740b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc27c0bc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc274144a30 0x7fc2740b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc27c0bc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fc274150690 con 0x7fc27409d2e0 2024-04-18T10:06:33.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc27c0bc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fc274150510 con 0x7fc2740ae6e0 2024-04-18T10:06:33.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc27c0bc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fc274150810 con 0x7fc274144a30 2024-04-18T10:06:33.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc279e58640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc27409d2e0 0x7fc274140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc279e58640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc27409d2e0 0x7fc274140da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40916/0 (socket says 172.21.15.62:40916) 2024-04-18T10:06:33.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc279e58640 1 -- 172.21.15.62:0/613092007 learned_addr learned my addr 172.21.15.62:0/613092007 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:33.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc27a659640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc274144a30 0x7fc2740b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc279e58640 1 -- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc274144a30 msgr2=0x7fc2740b2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc279e58640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc274144a30 0x7fc2740b2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc279e58640 1 -- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2740ae6e0 msgr2=0x7fc2740b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc279657640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2740ae6e0 0x7fc2740b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc279e58640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2740ae6e0 0x7fc2740b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc279e58640 1 -- 172.21.15.62:0/613092007 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc27413ef70 con 0x7fc27409d2e0 2024-04-18T10:06:33.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc279e58640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc27409d2e0 0x7fc274140da0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc27001a100 tx=0x7fc270019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:33.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc26a7fc640 1 -- 172.21.15.62:0/613092007 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2700b0040 con 0x7fc27409d2e0 2024-04-18T10:06:33.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc26a7fc640 1 -- 172.21.15.62:0/613092007 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2700b9070 con 0x7fc27409d2e0 2024-04-18T10:06:33.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc26a7fc640 1 -- 172.21.15.62:0/613092007 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2700b4880 con 0x7fc27409d2e0 2024-04-18T10:06:33.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc27c0bc640 1 -- 172.21.15.62:0/613092007 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc274140340 con 0x7fc27409d2e0 2024-04-18T10:06:33.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc27c0bc640 1 -- 172.21.15.62:0/613092007 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc27413f580 con 0x7fc27409d2e0 2024-04-18T10:06:33.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc27c0bc640 1 -- 172.21.15.62:0/613092007 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc274068b20 con 0x7fc27409d2e0 2024-04-18T10:06:33.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc26a7fc640 1 -- 172.21.15.62:0/613092007 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc2700b4a20 con 0x7fc27409d2e0 2024-04-18T10:06:33.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.433+0000 7fc26a7fc640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc2580683b0 0x7fc25806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.437+0000 7fc26a7fc640 1 -- 172.21.15.62:0/613092007 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7fc270120550 con 0x7fc27409d2e0 2024-04-18T10:06:33.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.437+0000 7fc279657640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc2580683b0 0x7fc25806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.437+0000 7fc279657640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc2580683b0 0x7fc25806a870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc2741dcbf0 tx=0x7fc264008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:33.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.441+0000 7fc26a7fc640 1 -- 172.21.15.62:0/613092007 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc270016150 con 0x7fc27409d2e0 2024-04-18T10:06:33.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.657+0000 7fc27c0bc640 1 -- 172.21.15.62:0/613092007 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7fc2740b3850 con 0x7fc27409d2e0 2024-04-18T10:06:33.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.661+0000 7fc26a7fc640 1 -- 172.21.15.62:0/613092007 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7fc2700f0ed0 con 0x7fc27409d2e0 2024-04-18T10:06:33.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.665+0000 7fc27c0bc640 1 -- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc2580683b0 msgr2=0x7fc25806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.665+0000 7fc27c0bc640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc2580683b0 0x7fc25806a870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc2741dcbf0 tx=0x7fc264008040 comp rx=0 tx=0).stop 2024-04-18T10:06:33.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.665+0000 7fc27c0bc640 1 -- 172.21.15.62:0/613092007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc27409d2e0 msgr2=0x7fc274140da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.665+0000 7fc27c0bc640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc27409d2e0 0x7fc274140da0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc27001a100 tx=0x7fc270019270 comp rx=0 tx=0).stop 2024-04-18T10:06:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.665+0000 7fc27c0bc640 1 -- 172.21.15.62:0/613092007 shutdown_connections 2024-04-18T10:06:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.665+0000 7fc27c0bc640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fc2580683b0 0x7fc25806a870 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.665+0000 7fc27c0bc640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc274144a30 0x7fc2740b2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.665+0000 7fc27c0bc640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2740ae6e0 0x7fc2740b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.665+0000 7fc27c0bc640 1 --2- 172.21.15.62:0/613092007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc27409d2e0 0x7fc274140da0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.665+0000 7fc27c0bc640 1 -- 172.21.15.62:0/613092007 >> 172.21.15.62:0/613092007 conn(0x7fc2740b9420 msgr2=0x7fc274096ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.665+0000 7fc27c0bc640 1 -- 172.21.15.62:0/613092007 shutdown_connections 2024-04-18T10:06:33.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.665+0000 7fc27c0bc640 1 -- 172.21.15.62:0/613092007 wait complete. 2024-04-18T10:06:33.680 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-18T10:06:33.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f800e08d640 1 Processor -- start 2024-04-18T10:06:33.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f800e08d640 1 -- start start 2024-04-18T10:06:33.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f800e08d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f800809c4d0 0x7f800809ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f800e08d640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f800809e2d0 0x7f8008142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f800e08d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f800809d430 con 0x7f800809b620 2024-04-18T10:06:33.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f800e08d640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f800806eb30 con 0x7f800809c4d0 2024-04-18T10:06:33.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f800e08d640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f800809dea0 con 0x7f800809e2d0 2024-04-18T10:06:33.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f8006ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f800809c4d0 0x7f800809ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f8006ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f800809c4d0 0x7f800809ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40926/0 (socket says 172.21.15.62:40926) 2024-04-18T10:06:33.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f8006ffd640 1 -- 172.21.15.62:0/3317941311 learned_addr learned my addr 172.21.15.62:0/3317941311 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:33.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f8007fff640 1 --2- 172.21.15.62:0/3317941311 >> v2:172.21.15.145:3300/0 conn(0x7f800809e2d0 0x7f8008142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f8006ffd640 1 -- 172.21.15.62:0/3317941311 >> v2:172.21.15.145:3300/0 conn(0x7f800809e2d0 msgr2=0x7f8008142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:33.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f8006ffd640 1 --2- 172.21.15.62:0/3317941311 >> v2:172.21.15.145:3300/0 conn(0x7f800809e2d0 0x7f8008142ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f8006ffd640 1 -- 172.21.15.62:0/3317941311 >> v1:172.21.15.145:6789/0 conn(0x7f800809b620 legacy=0x7f80081492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:33.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.793+0000 7f8006ffd640 1 -- 172.21.15.62:0/3317941311 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800809d6a0 con 0x7f800809c4d0 2024-04-18T10:06:33.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f8006ffd640 1 --2- 172.21.15.62:0/3317941311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f800809c4d0 0x7f800809ba00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc009870 tx=0x7f7ffc01c190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895307162ea2a6c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:33.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f80067fc640 1 -- 172.21.15.62:0/3317941311 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0b0070 con 0x7f800809c4d0 2024-04-18T10:06:33.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f80067fc640 1 -- 172.21.15.62:0/3317941311 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ffc0ab600 con 0x7f800809c4d0 2024-04-18T10:06:33.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f80067fc640 1 -- 172.21.15.62:0/3317941311 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0ab9d0 con 0x7f800809c4d0 2024-04-18T10:06:33.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 -- 172.21.15.62:0/3317941311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f800809c4d0 msgr2=0x7f800809ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 --2- 172.21.15.62:0/3317941311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f800809c4d0 0x7f800809ba00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc009870 tx=0x7f7ffc01c190 comp rx=0 tx=0).stop 2024-04-18T10:06:33.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 -- 172.21.15.62:0/3317941311 shutdown_connections 2024-04-18T10:06:33.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 --2- 172.21.15.62:0/3317941311 >> v2:172.21.15.145:3300/0 conn(0x7f800809e2d0 0x7f8008142ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 --2- 172.21.15.62:0/3317941311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f800809c4d0 0x7f800809ba00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 -- 172.21.15.62:0/3317941311 >> 172.21.15.62:0/3317941311 conn(0x7f80080b9420 msgr2=0x7f80080b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:33.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 -- 172.21.15.62:0/3317941311 shutdown_connections 2024-04-18T10:06:33.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 -- 172.21.15.62:0/3317941311 wait complete. 2024-04-18T10:06:33.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 Processor -- start 2024-04-18T10:06:33.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 -- start start 2024-04-18T10:06:33.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f800809b620 0x7f80080b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f800809c4d0 0x7f80081e1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f800809e2d0 0x7f80081e20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8008152b90 con 0x7f800809b620 2024-04-18T10:06:33.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f8008152a10 con 0x7f800809e2d0 2024-04-18T10:06:33.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f800e08d640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8008152d10 con 0x7f800809c4d0 2024-04-18T10:06:33.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f80077fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f800809b620 0x7f80080b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f80077fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f800809b620 0x7f80080b7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40940/0 (socket says 172.21.15.62:40940) 2024-04-18T10:06:33.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.797+0000 7f80077fe640 1 -- 172.21.15.62:0/22993971 learned_addr learned my addr 172.21.15.62:0/22993971 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:33.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f8006ffd640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f800809c4d0 0x7f80081e1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f80077fe640 1 -- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f800809c4d0 msgr2=0x7f80081e1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f80077fe640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f800809c4d0 0x7f80081e1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f8007fff640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f800809e2d0 0x7f80081e20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f80077fe640 1 -- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f800809e2d0 msgr2=0x7f80081e20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:33.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f80077fe640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f800809e2d0 0x7f80081e20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:33.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f80077fe640 1 -- 172.21.15.62:0/22993971 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80080b0f80 con 0x7f800809b620 2024-04-18T10:06:33.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f80077fe640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f800809b620 0x7f80080b7fb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800c450 tx=0x7f7ff800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:33.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f7feffff640 1 -- 172.21.15.62:0/22993971 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8012830 con 0x7f800809b620 2024-04-18T10:06:33.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f7feffff640 1 -- 172.21.15.62:0/22993971 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ff8017070 con 0x7f800809b620 2024-04-18T10:06:33.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f800e08d640 1 -- 172.21.15.62:0/22993971 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80080b7aa0 con 0x7f800809b620 2024-04-18T10:06:33.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f7feffff640 1 -- 172.21.15.62:0/22993971 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8012c30 con 0x7f800809b620 2024-04-18T10:06:33.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f800e08d640 1 -- 172.21.15.62:0/22993971 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f800813f6d0 con 0x7f800809b620 2024-04-18T10:06:33.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f800e08d640 1 -- 172.21.15.62:0/22993971 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8008068b20 con 0x7f800809b620 2024-04-18T10:06:33.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f7feffff640 1 -- 172.21.15.62:0/22993971 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7ff801c460 con 0x7f800809b620 2024-04-18T10:06:33.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f7feffff640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ff4068380 0x7f7ff406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:33.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f7feffff640 1 -- 172.21.15.62:0/22993971 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f7ff8086420 con 0x7f800809b620 2024-04-18T10:06:33.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f8006ffd640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ff4068380 0x7f7ff406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:33.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.801+0000 7f8006ffd640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ff4068380 0x7f7ff406a840 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0ad000 tx=0x7f7ffc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:33.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:33.805+0000 7f7feffff640 1 -- 172.21.15.62:0/22993971 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7ff8056d40 con 0x7f800809b620 2024-04-18T10:06:34.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.021+0000 7f800e08d640 1 -- 172.21.15.62:0/22993971 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f800809e050 con 0x7f800809b620 2024-04-18T10:06:34.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f7feffff640 1 -- 172.21.15.62:0/22993971 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f7ff8056640 con 0x7f800809b620 2024-04-18T10:06:34.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f800e08d640 1 -- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ff4068380 msgr2=0x7f7ff406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f800e08d640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ff4068380 0x7f7ff406a840 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc0ad000 tx=0x7f7ffc008040 comp rx=0 tx=0).stop 2024-04-18T10:06:34.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f800e08d640 1 -- 172.21.15.62:0/22993971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f800809b620 msgr2=0x7f80080b7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f800e08d640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f800809b620 0x7f80080b7fb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7ff800c450 tx=0x7f7ff800c910 comp rx=0 tx=0).stop 2024-04-18T10:06:34.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f800e08d640 1 -- 172.21.15.62:0/22993971 shutdown_connections 2024-04-18T10:06:34.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f800e08d640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ff4068380 0x7f7ff406a840 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f800e08d640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f800809e2d0 0x7f80081e20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f800e08d640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f800809c4d0 0x7f80081e1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f800e08d640 1 --2- 172.21.15.62:0/22993971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f800809b620 0x7f80080b7fb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f800e08d640 1 -- 172.21.15.62:0/22993971 >> 172.21.15.62:0/22993971 conn(0x7f80080b9420 msgr2=0x7f8008144ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:34.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f800e08d640 1 -- 172.21.15.62:0/22993971 shutdown_connections 2024-04-18T10:06:34.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.029+0000 7f800e08d640 1 -- 172.21.15.62:0/22993971 wait complete. 2024-04-18T10:06:34.046 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-18T10:06:34.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a200be640 1 Processor -- start 2024-04-18T10:06:34.168 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a200be640 1 -- start start 2024-04-18T10:06:34.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a200be640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a180ba9f0 0x7f9a1809b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a200be640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9a1814b1f0 0x7f9a181418a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a200be640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9a18141de0 0x7f9a18147c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a200be640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f9a1814b760 con 0x7f9a1814b1f0 2024-04-18T10:06:34.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a200be640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f9a1814ba40 con 0x7f9a18141de0 2024-04-18T10:06:34.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a200be640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f9a1814bd20 con 0x7f9a180ba9f0 2024-04-18T10:06:34.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a1d659640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9a1814b1f0 0x7f9a181418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a1d659640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9a1814b1f0 0x7f9a181418a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40948/0 (socket says 172.21.15.62:40948) 2024-04-18T10:06:34.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a1d659640 1 -- 172.21.15.62:0/2535385952 learned_addr learned my addr 172.21.15.62:0/2535385952 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:34.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a1de5a640 1 --2- 172.21.15.62:0/2535385952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a180ba9f0 0x7f9a1809b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a1d659640 1 -- 172.21.15.62:0/2535385952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a180ba9f0 msgr2=0x7f9a1809b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:34.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a1d659640 1 --2- 172.21.15.62:0/2535385952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a180ba9f0 0x7f9a1809b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a1d659640 1 -- 172.21.15.62:0/2535385952 >> v2:172.21.15.145:3300/0 conn(0x7f9a18141de0 msgr2=0x7f9a18147c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:34.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a1e65b640 1 --2- 172.21.15.62:0/2535385952 >> v2:172.21.15.145:3300/0 conn(0x7f9a18141de0 0x7f9a18147c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a1d659640 1 --2- 172.21.15.62:0/2535385952 >> v2:172.21.15.145:3300/0 conn(0x7f9a18141de0 0x7f9a18147c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.169+0000 7f9a1d659640 1 -- 172.21.15.62:0/2535385952 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a1809cc40 con 0x7f9a1814b1f0 2024-04-18T10:06:34.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1d659640 1 --2- 172.21.15.62:0/2535385952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1814b1f0 0x7f9a181418a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9a1814bc50 tx=0x7f9a0801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab9989992bf02b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:34.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1ce58640 1 -- 172.21.15.62:0/2535385952 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080b0070 con 0x7f9a1814b1f0 2024-04-18T10:06:34.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1ce58640 1 -- 172.21.15.62:0/2535385952 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a080ab600 con 0x7f9a1814b1f0 2024-04-18T10:06:34.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1ce58640 1 -- 172.21.15.62:0/2535385952 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080ab9d0 con 0x7f9a1814b1f0 2024-04-18T10:06:34.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 -- 172.21.15.62:0/2535385952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1814b1f0 msgr2=0x7f9a181418a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 --2- 172.21.15.62:0/2535385952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1814b1f0 0x7f9a181418a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9a1814bc50 tx=0x7f9a0801c490 comp rx=0 tx=0).stop 2024-04-18T10:06:34.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 -- 172.21.15.62:0/2535385952 shutdown_connections 2024-04-18T10:06:34.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 --2- 172.21.15.62:0/2535385952 >> v2:172.21.15.145:3300/0 conn(0x7f9a18141de0 0x7f9a18147c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 --2- 172.21.15.62:0/2535385952 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1814b1f0 0x7f9a181418a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 --2- 172.21.15.62:0/2535385952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a180ba9f0 0x7f9a1809b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 -- 172.21.15.62:0/2535385952 >> 172.21.15.62:0/2535385952 conn(0x7f9a18067ea0 msgr2=0x7f9a180ba080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:34.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 -- 172.21.15.62:0/2535385952 shutdown_connections 2024-04-18T10:06:34.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 -- 172.21.15.62:0/2535385952 wait complete. 2024-04-18T10:06:34.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 Processor -- start 2024-04-18T10:06:34.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 -- start start 2024-04-18T10:06:34.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a180ba9f0 0x7f9a180bba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a18141de0 0x7f9a180a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1814b1f0 0x7f9a181e3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1806cd10 con 0x7f9a1814b1f0 2024-04-18T10:06:34.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1806cb90 con 0x7f9a18141de0 2024-04-18T10:06:34.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a200be640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f9a1806ce90 con 0x7f9a180ba9f0 2024-04-18T10:06:34.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1e65b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1814b1f0 0x7f9a181e3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1e65b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1814b1f0 0x7f9a181e3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40960/0 (socket says 172.21.15.62:40960) 2024-04-18T10:06:34.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1e65b640 1 -- 172.21.15.62:0/1751576179 learned_addr learned my addr 172.21.15.62:0/1751576179 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:34.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1de5a640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a180ba9f0 0x7f9a180bba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1d659640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a18141de0 0x7f9a180a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1e65b640 1 -- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a180ba9f0 msgr2=0x7f9a180bba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1e65b640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a180ba9f0 0x7f9a180bba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1e65b640 1 -- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a18141de0 msgr2=0x7f9a180a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1e65b640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a18141de0 0x7f9a180a1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1e65b640 1 -- 172.21.15.62:0/1751576179 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a180bcbe0 con 0x7f9a1814b1f0 2024-04-18T10:06:34.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1de5a640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a180ba9f0 0x7f9a180bba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:34.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.173+0000 7f9a1e65b640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1814b1f0 0x7f9a181e3a90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9a1400c7f0 tx=0x7f9a1400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:34.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.177+0000 7f9a0e7fc640 1 -- 172.21.15.62:0/1751576179 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14015070 con 0x7f9a1814b1f0 2024-04-18T10:06:34.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.177+0000 7f9a0e7fc640 1 -- 172.21.15.62:0/1751576179 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a14004220 con 0x7f9a1814b1f0 2024-04-18T10:06:34.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.177+0000 7f9a200be640 1 -- 172.21.15.62:0/1751576179 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a180bcec0 con 0x7f9a1814b1f0 2024-04-18T10:06:34.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.177+0000 7f9a0e7fc640 1 -- 172.21.15.62:0/1751576179 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a14003620 con 0x7f9a1814b1f0 2024-04-18T10:06:34.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.177+0000 7f9a200be640 1 -- 172.21.15.62:0/1751576179 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a1809ef60 con 0x7f9a1814b1f0 2024-04-18T10:06:34.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.177+0000 7f9a200be640 1 -- 172.21.15.62:0/1751576179 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a18068b20 con 0x7f9a1814b1f0 2024-04-18T10:06:34.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.177+0000 7f9a0e7fc640 1 -- 172.21.15.62:0/1751576179 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9a140039d0 con 0x7f9a1814b1f0 2024-04-18T10:06:34.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.177+0000 7f9a0e7fc640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f99fc068380 0x7f99fc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.177+0000 7f9a0e7fc640 1 -- 172.21.15.62:0/1751576179 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f9a14086680 con 0x7f9a1814b1f0 2024-04-18T10:06:34.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.177+0000 7f9a1de5a640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f99fc068380 0x7f99fc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.177+0000 7f9a1de5a640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f99fc068380 0x7f99fc06a840 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9a10007da0 tx=0x7f9a100072f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:34.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.181+0000 7f9a0e7fc640 1 -- 172.21.15.62:0/1751576179 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9a140570e0 con 0x7f9a1814b1f0 2024-04-18T10:06:34.394 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.393+0000 7f9a200be640 1 -- 172.21.15.62:0/1751576179 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f9a180b9e10 con 0x7f9a1814b1f0 2024-04-18T10:06:34.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.397+0000 7f9a0e7fc640 1 -- 172.21.15.62:0/1751576179 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f9a140569e0 con 0x7f9a1814b1f0 2024-04-18T10:06:34.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.401+0000 7f9a200be640 1 -- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f99fc068380 msgr2=0x7f99fc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.401+0000 7f9a200be640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f99fc068380 0x7f99fc06a840 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9a10007da0 tx=0x7f9a100072f0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.401+0000 7f9a200be640 1 -- 172.21.15.62:0/1751576179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1814b1f0 msgr2=0x7f9a181e3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.401+0000 7f9a200be640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1814b1f0 0x7f9a181e3a90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9a1400c7f0 tx=0x7f9a1400ccb0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.401+0000 7f9a200be640 1 -- 172.21.15.62:0/1751576179 shutdown_connections 2024-04-18T10:06:34.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.401+0000 7f9a200be640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f99fc068380 0x7f99fc06a840 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.401+0000 7f9a200be640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a1814b1f0 0x7f9a181e3a90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.401+0000 7f9a200be640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a18141de0 0x7f9a180a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.401+0000 7f9a200be640 1 --2- 172.21.15.62:0/1751576179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a180ba9f0 0x7f9a180bba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.401+0000 7f9a200be640 1 -- 172.21.15.62:0/1751576179 >> 172.21.15.62:0/1751576179 conn(0x7f9a18067ea0 msgr2=0x7f9a181408e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:34.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.401+0000 7f9a200be640 1 -- 172.21.15.62:0/1751576179 shutdown_connections 2024-04-18T10:06:34.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.401+0000 7f9a200be640 1 -- 172.21.15.62:0/1751576179 wait complete. 2024-04-18T10:06:34.416 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-18T10:06:34.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f180e4db640 1 Processor -- start 2024-04-18T10:06:34.528 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f180e4db640 1 -- start start 2024-04-18T10:06:34.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f180e4db640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1808148150 0x7f18080b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f180e4db640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f18080b0040 0x7f180809c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f180e4db640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18080b6e50 con 0x7f18080b0b70 2024-04-18T10:06:34.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f180e4db640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f18080af560 con 0x7f18080b0040 2024-04-18T10:06:34.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f180e4db640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f18080af840 con 0x7f1808148150 2024-04-18T10:06:34.529 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f180ca78640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f18080b0040 0x7f180809c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f1807fff640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1808148150 0x7f18080b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f180ca78640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f18080b0040 0x7f180809c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.62:47318/0 (socket says 172.21.15.62:47318) 2024-04-18T10:06:34.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f180ca78640 1 -- 172.21.15.62:0/816246983 learned_addr learned my addr 172.21.15.62:0/816246983 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:34.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f1807fff640 1 -- 172.21.15.62:0/816246983 >> v2:172.21.15.145:3300/0 conn(0x7f18080b0040 msgr2=0x7f180809c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:34.530 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f1807fff640 1 --2- 172.21.15.62:0/816246983 >> v2:172.21.15.145:3300/0 conn(0x7f18080b0040 0x7f180809c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f1807fff640 1 -- 172.21.15.62:0/816246983 >> v1:172.21.15.145:6789/0 conn(0x7f18080b0b70 legacy=0x7f1808148530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f1807fff640 1 -- 172.21.15.62:0/816246983 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18080afb20 con 0x7f1808148150 2024-04-18T10:06:34.531 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.529+0000 7f180ca78640 1 --2- 172.21.15.62:0/816246983 >> v2:172.21.15.145:3300/0 conn(0x7f18080b0040 0x7f180809c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:06:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f1807fff640 1 --2- 172.21.15.62:0/816246983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1808148150 0x7f18080b4f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f17f80097c0 tx=0x7f17f801c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=651d91c3dc24dcf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f1806ffd640 1 -- 172.21.15.62:0/816246983 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f80b0070 con 0x7f1808148150 2024-04-18T10:06:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f1806ffd640 1 -- 172.21.15.62:0/816246983 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f80ab600 con 0x7f1808148150 2024-04-18T10:06:34.532 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f1806ffd640 1 -- 172.21.15.62:0/816246983 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f80ab9d0 con 0x7f1808148150 2024-04-18T10:06:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f180e4db640 1 -- 172.21.15.62:0/816246983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1808148150 msgr2=0x7f18080b4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f180e4db640 1 --2- 172.21.15.62:0/816246983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1808148150 0x7f18080b4f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f17f80097c0 tx=0x7f17f801c380 comp rx=0 tx=0).stop 2024-04-18T10:06:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f180e4db640 1 -- 172.21.15.62:0/816246983 shutdown_connections 2024-04-18T10:06:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f180e4db640 1 --2- 172.21.15.62:0/816246983 >> v2:172.21.15.145:3300/0 conn(0x7f18080b0040 0x7f180809c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f180e4db640 1 --2- 172.21.15.62:0/816246983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1808148150 0x7f18080b4f40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f180e4db640 1 -- 172.21.15.62:0/816246983 >> 172.21.15.62:0/816246983 conn(0x7f18080b7040 msgr2=0x7f18080b7450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:34.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f180e4db640 1 -- 172.21.15.62:0/816246983 shutdown_connections 2024-04-18T10:06:34.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f180e4db640 1 -- 172.21.15.62:0/816246983 wait complete. 2024-04-18T10:06:34.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f180e4db640 1 Processor -- start 2024-04-18T10:06:34.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.533+0000 7f180e4db640 1 -- start start 2024-04-18T10:06:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f180e4db640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18080b0040 0x7f18081e4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f180e4db640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1808148150 0x7f1808142d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f180e4db640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1808146b50 0x7f18081e0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f180e4db640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1808151dd0 con 0x7f1808148150 2024-04-18T10:06:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f180e4db640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f1808151c50 con 0x7f18080b0040 2024-04-18T10:06:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f180e4db640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f1808151f50 con 0x7f1808146b50 2024-04-18T10:06:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f18077fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1808148150 0x7f1808142d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f18077fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1808148150 0x7f1808142d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40964/0 (socket says 172.21.15.62:40964) 2024-04-18T10:06:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f18077fe640 1 -- 172.21.15.62:0/2820983588 learned_addr learned my addr 172.21.15.62:0/2820983588 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f1807fff640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18080b0040 0x7f18081e4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f180ca78640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1808146b50 0x7f18081e0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f18077fe640 1 -- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1808146b50 msgr2=0x7f18081e0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f18077fe640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1808146b50 0x7f18081e0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f18077fe640 1 -- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18080b0040 msgr2=0x7f18081e4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f18077fe640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18080b0040 0x7f18081e4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f18077fe640 1 -- 172.21.15.62:0/2820983588 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1808146070 con 0x7f1808148150 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f180ca78640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1808146b50 0x7f18081e0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f1807fff640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18080b0040 0x7f18081e4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f18077fe640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1808148150 0x7f1808142d80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00c420 tx=0x7f17fc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f1804ff9640 1 -- 172.21.15.62:0/2820983588 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc0052d0 con 0x7f1808148150 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f1804ff9640 1 -- 172.21.15.62:0/2820983588 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17fc005470 con 0x7f1808148150 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f1804ff9640 1 -- 172.21.15.62:0/2820983588 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc007260 con 0x7f1808148150 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f180e4db640 1 -- 172.21.15.62:0/2820983588 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1808146350 con 0x7f1808148150 2024-04-18T10:06:34.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f180e4db640 1 -- 172.21.15.62:0/2820983588 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18081427f0 con 0x7f1808148150 2024-04-18T10:06:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f180e4db640 1 -- 172.21.15.62:0/2820983588 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1808068390 con 0x7f1808148150 2024-04-18T10:06:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f1804ff9640 1 -- 172.21.15.62:0/2820983588 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f17fc007400 con 0x7f1808148150 2024-04-18T10:06:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f1804ff9640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f17f4068380 0x7f17f406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f1804ff9640 1 -- 172.21.15.62:0/2820983588 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f17fc085b30 con 0x7f1808148150 2024-04-18T10:06:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.537+0000 7f1807fff640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f17f4068380 0x7f17f406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.541+0000 7f1807fff640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f17f4068380 0x7f17f406a840 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f17f80ad000 tx=0x7f17f80a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:34.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.541+0000 7f1804ff9640 1 -- 172.21.15.62:0/2820983588 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f17fc056440 con 0x7f1808148150 2024-04-18T10:06:34.754 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.753+0000 7f180e4db640 1 -- 172.21.15.62:0/2820983588 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f1808069230 con 0x7f1808148150 2024-04-18T10:06:34.758 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.757+0000 7f1804ff9640 1 -- 172.21.15.62:0/2820983588 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f17fc055d40 con 0x7f1808148150 2024-04-18T10:06:34.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.761+0000 7f180e4db640 1 -- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f17f4068380 msgr2=0x7f17f406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.761+0000 7f180e4db640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f17f4068380 0x7f17f406a840 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f17f80ad000 tx=0x7f17f80a6000 comp rx=0 tx=0).stop 2024-04-18T10:06:34.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.761+0000 7f180e4db640 1 -- 172.21.15.62:0/2820983588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1808148150 msgr2=0x7f1808142d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.761+0000 7f180e4db640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1808148150 0x7f1808142d80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f17fc00c420 tx=0x7f17fc00c8e0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.761+0000 7f180e4db640 1 -- 172.21.15.62:0/2820983588 shutdown_connections 2024-04-18T10:06:34.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.761+0000 7f180e4db640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f17f4068380 0x7f17f406a840 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.761+0000 7f180e4db640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1808146b50 0x7f18081e0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.761+0000 7f180e4db640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1808148150 0x7f1808142d80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.761+0000 7f180e4db640 1 --2- 172.21.15.62:0/2820983588 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18080b0040 0x7f18081e4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.761+0000 7f180e4db640 1 -- 172.21.15.62:0/2820983588 >> 172.21.15.62:0/2820983588 conn(0x7f18080b7040 msgr2=0x7f180809bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:34.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.761+0000 7f180e4db640 1 -- 172.21.15.62:0/2820983588 shutdown_connections 2024-04-18T10:06:34.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.761+0000 7f180e4db640 1 -- 172.21.15.62:0/2820983588 wait complete. 2024-04-18T10:06:34.777 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-18T10:06:34.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f56725f3640 1 Processor -- start 2024-04-18T10:06:34.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f56725f3640 1 -- start start 2024-04-18T10:06:34.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f56725f3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f566c06ec70 0x7f566c0b98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f56725f3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f566c06c530 0x7f566c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f56725f3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f566c0b53e0 0x7f566c0b5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f56725f3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f566c099f20 con 0x7f566c0b53e0 2024-04-18T10:06:34.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f56725f3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f566c06c9b0 con 0x7f566c06c530 2024-04-18T10:06:34.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f56725f3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f566c0b6ea0 con 0x7f566c06ec70 2024-04-18T10:06:34.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f5670b90640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f566c0b53e0 0x7f566c0b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f5670b90640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f566c0b53e0 0x7f566c0b5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40970/0 (socket says 172.21.15.62:40970) 2024-04-18T10:06:34.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f5670b90640 1 -- 172.21.15.62:0/50187289 learned_addr learned my addr 172.21.15.62:0/50187289 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:34.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f566bfff640 1 --2- 172.21.15.62:0/50187289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f566c06ec70 0x7f566c0b98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f566b7fe640 1 --2- 172.21.15.62:0/50187289 >> v2:172.21.15.145:3300/0 conn(0x7f566c06c530 0x7f566c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f5670b90640 1 -- 172.21.15.62:0/50187289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f566c06ec70 msgr2=0x7f566c0b98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:34.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f5670b90640 1 --2- 172.21.15.62:0/50187289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f566c06ec70 0x7f566c0b98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f5670b90640 1 -- 172.21.15.62:0/50187289 >> v2:172.21.15.145:3300/0 conn(0x7f566c06c530 msgr2=0x7f566c0b4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:34.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f5670b90640 1 --2- 172.21.15.62:0/50187289 >> v2:172.21.15.145:3300/0 conn(0x7f566c06c530 0x7f566c0b4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.901+0000 7f5670b90640 1 -- 172.21.15.62:0/50187289 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566c06f050 con 0x7f566c0b53e0 2024-04-18T10:06:34.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f5670b90640 1 --2- 172.21.15.62:0/50187289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f566c0b53e0 0x7f566c0b5870 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f566c068750 tx=0x7f565401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e7ed64e58605b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:34.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f566affd640 1 -- 172.21.15.62:0/50187289 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56540b0070 con 0x7f566c0b53e0 2024-04-18T10:06:34.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f566affd640 1 -- 172.21.15.62:0/50187289 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56540ab600 con 0x7f566c0b53e0 2024-04-18T10:06:34.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f566affd640 1 -- 172.21.15.62:0/50187289 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56540ab9d0 con 0x7f566c0b53e0 2024-04-18T10:06:34.906 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f56725f3640 1 -- 172.21.15.62:0/50187289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f566c0b53e0 msgr2=0x7f566c0b5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f56725f3640 1 --2- 172.21.15.62:0/50187289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f566c0b53e0 0x7f566c0b5870 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f566c068750 tx=0x7f565401c400 comp rx=0 tx=0).stop 2024-04-18T10:06:34.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f56725f3640 1 -- 172.21.15.62:0/50187289 shutdown_connections 2024-04-18T10:06:34.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f56725f3640 1 --2- 172.21.15.62:0/50187289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f566c0b53e0 0x7f566c0b5870 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f56725f3640 1 --2- 172.21.15.62:0/50187289 >> v2:172.21.15.145:3300/0 conn(0x7f566c06c530 0x7f566c0b4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f56725f3640 1 --2- 172.21.15.62:0/50187289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f566c06ec70 0x7f566c0b98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f56725f3640 1 -- 172.21.15.62:0/50187289 >> 172.21.15.62:0/50187289 conn(0x7f566c06f6d0 msgr2=0x7f566c06e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:34.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f56725f3640 1 -- 172.21.15.62:0/50187289 shutdown_connections 2024-04-18T10:06:34.907 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.905+0000 7f56725f3640 1 -- 172.21.15.62:0/50187289 wait complete. 2024-04-18T10:06:34.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f56725f3640 1 Processor -- start 2024-04-18T10:06:34.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f56725f3640 1 -- start start 2024-04-18T10:06:34.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f56725f3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f566c06c530 0x7f566c151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f56725f3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f566c06ec70 0x7f566c14e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f56725f3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f566c0b53e0 0x7f566c145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f56725f3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f566c0b7200 con 0x7f566c0b53e0 2024-04-18T10:06:34.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f56725f3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f566c0b8dd0 con 0x7f566c06c530 2024-04-18T10:06:34.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f56725f3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f566c0b82a0 con 0x7f566c06ec70 2024-04-18T10:06:34.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5670b90640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f566c0b53e0 0x7f566c145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.910 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5670b90640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f566c0b53e0 0x7f566c145540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:40986/0 (socket says 172.21.15.62:40986) 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5670b90640 1 -- 172.21.15.62:0/303457307 learned_addr learned my addr 172.21.15.62:0/303457307 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f566b7fe640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f566c06ec70 0x7f566c14e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f566bfff640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f566c06c530 0x7f566c151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5670b90640 1 -- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f566c06ec70 msgr2=0x7f566c14e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5670b90640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f566c06ec70 0x7f566c14e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5670b90640 1 -- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f566c06c530 msgr2=0x7f566c151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5670b90640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f566c06c530 0x7f566c151370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5670b90640 1 -- 172.21.15.62:0/303457307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566c09c0f0 con 0x7f566c0b53e0 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f566b7fe640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f566c06ec70 0x7f566c14e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f566bfff640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f566c06c530 0x7f566c151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5670b90640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f566c0b53e0 0x7f566c145540 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f565401aab0 tx=0x7f56540191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5668ff9640 1 -- 172.21.15.62:0/303457307 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56540b0040 con 0x7f566c0b53e0 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5668ff9640 1 -- 172.21.15.62:0/303457307 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56540b4420 con 0x7f566c0b53e0 2024-04-18T10:06:34.911 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5668ff9640 1 -- 172.21.15.62:0/303457307 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56540b47d0 con 0x7f566c0b53e0 2024-04-18T10:06:34.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f56725f3640 1 -- 172.21.15.62:0/303457307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f566c09cea0 con 0x7f566c0b53e0 2024-04-18T10:06:34.912 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f56725f3640 1 -- 172.21.15.62:0/303457307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f566c14d450 con 0x7f566c0b53e0 2024-04-18T10:06:34.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f56725f3640 1 -- 172.21.15.62:0/303457307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f566c09a990 con 0x7f566c0b53e0 2024-04-18T10:06:34.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.909+0000 7f5668ff9640 1 -- 172.21.15.62:0/303457307 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f56540b4970 con 0x7f566c0b53e0 2024-04-18T10:06:34.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.913+0000 7f5668ff9640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f564c0683b0 0x7f564c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:34.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.913+0000 7f5668ff9640 1 -- 172.21.15.62:0/303457307 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f56541201a0 con 0x7f566c0b53e0 2024-04-18T10:06:34.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.913+0000 7f566bfff640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f564c0683b0 0x7f564c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:34.913 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.913+0000 7f566bfff640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f564c0683b0 0x7f564c06a870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f565c00b9f0 tx=0x7f565c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:34.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:34.917+0000 7f5668ff9640 1 -- 172.21.15.62:0/303457307 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5654016110 con 0x7f566c0b53e0 2024-04-18T10:06:35.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.141+0000 7f56725f3640 1 -- 172.21.15.62:0/303457307 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f566c069230 con 0x7f566c0b53e0 2024-04-18T10:06:35.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.145+0000 7f5668ff9640 1 -- 172.21.15.62:0/303457307 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7f56540f0b40 con 0x7f566c0b53e0 2024-04-18T10:06:35.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.149+0000 7f56725f3640 1 -- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f564c0683b0 msgr2=0x7f564c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.149+0000 7f56725f3640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f564c0683b0 0x7f564c06a870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f565c00b9f0 tx=0x7f565c008040 comp rx=0 tx=0).stop 2024-04-18T10:06:35.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.149+0000 7f56725f3640 1 -- 172.21.15.62:0/303457307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f566c0b53e0 msgr2=0x7f566c145540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.149+0000 7f56725f3640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f566c0b53e0 0x7f566c145540 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f565401aab0 tx=0x7f56540191a0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.149+0000 7f56725f3640 1 -- 172.21.15.62:0/303457307 shutdown_connections 2024-04-18T10:06:35.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.149+0000 7f56725f3640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f564c0683b0 0x7f564c06a870 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.149+0000 7f56725f3640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f566c0b53e0 0x7f566c145540 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.149+0000 7f56725f3640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f566c06ec70 0x7f566c14e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.149+0000 7f56725f3640 1 --2- 172.21.15.62:0/303457307 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f566c06c530 0x7f566c151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.149+0000 7f56725f3640 1 -- 172.21.15.62:0/303457307 >> 172.21.15.62:0/303457307 conn(0x7f566c06f6d0 msgr2=0x7f566c14ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:35.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.149+0000 7f56725f3640 1 -- 172.21.15.62:0/303457307 shutdown_connections 2024-04-18T10:06:35.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.149+0000 7f56725f3640 1 -- 172.21.15.62:0/303457307 wait complete. 2024-04-18T10:06:35.165 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-18T10:06:35.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca9bfa640 1 Processor -- start 2024-04-18T10:06:35.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca9bfa640 1 -- start start 2024-04-18T10:06:35.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca9bfa640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faca409b620 0x7faca409ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca9bfa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faca409c4d0 0x7faca41492b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca9bfa640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faca409d430 con 0x7faca41497f0 2024-04-18T10:06:35.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca9bfa640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7faca406eb30 con 0x7faca409b620 2024-04-18T10:06:35.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca9bfa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7faca409dea0 con 0x7faca409c4d0 2024-04-18T10:06:35.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca37fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faca409b620 0x7faca409ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca2ffd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faca409c4d0 0x7faca41492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca37fe640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faca409b620 0x7faca409ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.62:47348/0 (socket says 172.21.15.62:47348) 2024-04-18T10:06:35.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca37fe640 1 -- 172.21.15.62:0/1327406279 learned_addr learned my addr 172.21.15.62:0/1327406279 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:35.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca2ffd640 1 --2- 172.21.15.62:0/1327406279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faca409c4d0 0x7faca41492b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:59684/0 (socket says 172.21.15.62:59684) 2024-04-18T10:06:35.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca37fe640 1 -- 172.21.15.62:0/1327406279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faca409c4d0 msgr2=0x7faca41492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:35.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca37fe640 1 --2- 172.21.15.62:0/1327406279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faca409c4d0 0x7faca41492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca37fe640 1 -- 172.21.15.62:0/1327406279 >> v1:172.21.15.145:6789/0 conn(0x7faca41497f0 legacy=0x7faca406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca37fe640 1 -- 172.21.15.62:0/1327406279 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faca409d6a0 con 0x7faca409b620 2024-04-18T10:06:35.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.281+0000 7faca2ffd640 1 --2- 172.21.15.62:0/1327406279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faca409c4d0 0x7faca41492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:06:35.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.285+0000 7faca37fe640 1 --2- 172.21.15.62:0/1327406279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faca409b620 0x7faca409ba00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fac98009870 tx=0x7fac9801c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85a6ce27fdb86433 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:35.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.285+0000 7faca27fc640 1 -- 172.21.15.62:0/1327406279 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac980b0070 con 0x7faca409b620 2024-04-18T10:06:35.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.285+0000 7faca27fc640 1 -- 172.21.15.62:0/1327406279 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac980ab600 con 0x7faca409b620 2024-04-18T10:06:35.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.285+0000 7faca27fc640 1 -- 172.21.15.62:0/1327406279 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac980ab9d0 con 0x7faca409b620 2024-04-18T10:06:35.286 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.285+0000 7faca9bfa640 1 -- 172.21.15.62:0/1327406279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faca409b620 msgr2=0x7faca409ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.285+0000 7faca9bfa640 1 --2- 172.21.15.62:0/1327406279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faca409b620 0x7faca409ba00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fac98009870 tx=0x7fac9801c4d0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.285+0000 7faca9bfa640 1 -- 172.21.15.62:0/1327406279 shutdown_connections 2024-04-18T10:06:35.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.285+0000 7faca9bfa640 1 --2- 172.21.15.62:0/1327406279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faca409c4d0 0x7faca41492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.285+0000 7faca9bfa640 1 --2- 172.21.15.62:0/1327406279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faca409b620 0x7faca409ba00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.285+0000 7faca9bfa640 1 -- 172.21.15.62:0/1327406279 >> 172.21.15.62:0/1327406279 conn(0x7faca40b9420 msgr2=0x7faca40b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:35.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.285+0000 7faca9bfa640 1 -- 172.21.15.62:0/1327406279 shutdown_connections 2024-04-18T10:06:35.287 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.285+0000 7faca9bfa640 1 -- 172.21.15.62:0/1327406279 wait complete. 2024-04-18T10:06:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca9bfa640 1 Processor -- start 2024-04-18T10:06:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca9bfa640 1 -- start start 2024-04-18T10:06:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca9bfa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faca409b620 0x7faca413ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca9bfa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faca409c4d0 0x7faca40b7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca9bfa640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faca41497f0 0x7faca414e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca9bfa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7faca4152bf0 con 0x7faca409c4d0 2024-04-18T10:06:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca9bfa640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7faca4152a70 con 0x7faca41497f0 2024-04-18T10:06:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca9bfa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7faca4152d70 con 0x7faca409b620 2024-04-18T10:06:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faca409c4d0 0x7faca40b7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca2ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faca409c4d0 0x7faca40b7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41002/0 (socket says 172.21.15.62:41002) 2024-04-18T10:06:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca2ffd640 1 -- 172.21.15.62:0/654200787 learned_addr learned my addr 172.21.15.62:0/654200787 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca37fe640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faca409b620 0x7faca413ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca3fff640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faca41497f0 0x7faca414e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca2ffd640 1 -- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faca409b620 msgr2=0x7faca413ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca2ffd640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faca409b620 0x7faca413ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca2ffd640 1 -- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faca41497f0 msgr2=0x7faca414e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca2ffd640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faca41497f0 0x7faca414e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca2ffd640 1 -- 172.21.15.62:0/654200787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faca41e0eb0 con 0x7faca409c4d0 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca3fff640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faca41497f0 0x7faca414e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca2ffd640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faca409c4d0 0x7faca40b7cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fac9400e9d0 tx=0x7fac9400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7fac8bfff640 1 -- 172.21.15.62:0/654200787 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac94017070 con 0x7faca409c4d0 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7fac8bfff640 1 -- 172.21.15.62:0/654200787 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac9400ca60 con 0x7faca409c4d0 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca9bfa640 1 -- 172.21.15.62:0/654200787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faca41e1150 con 0x7faca409c4d0 2024-04-18T10:06:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7fac8bfff640 1 -- 172.21.15.62:0/654200787 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9400ce50 con 0x7faca409c4d0 2024-04-18T10:06:35.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca9bfa640 1 -- 172.21.15.62:0/654200787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faca4143860 con 0x7faca409c4d0 2024-04-18T10:06:35.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca9bfa640 1 -- 172.21.15.62:0/654200787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faca41e0ca0 con 0x7faca409c4d0 2024-04-18T10:06:35.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7fac8bfff640 1 -- 172.21.15.62:0/654200787 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fac9401a070 con 0x7faca409c4d0 2024-04-18T10:06:35.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7fac8bfff640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fac90068520 0x7fac9006a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7fac8bfff640 1 -- 172.21.15.62:0/654200787 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7fac94086230 con 0x7faca409c4d0 2024-04-18T10:06:35.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.289+0000 7faca37fe640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fac90068520 0x7fac9006a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.293 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.293+0000 7faca37fe640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fac90068520 0x7fac9006a9e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fac980ad000 tx=0x7fac980a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:35.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.293+0000 7fac8bfff640 1 -- 172.21.15.62:0/654200787 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fac94056b30 con 0x7faca409c4d0 2024-04-18T10:06:35.516 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.517+0000 7faca9bfa640 1 -- 172.21.15.62:0/654200787 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7faca4068b20 con 0x7faca409c4d0 2024-04-18T10:06:35.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7fac8bfff640 1 -- 172.21.15.62:0/654200787 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7fac94056430 con 0x7faca409c4d0 2024-04-18T10:06:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7faca9bfa640 1 -- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fac90068520 msgr2=0x7fac9006a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7faca9bfa640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fac90068520 0x7fac9006a9e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fac980ad000 tx=0x7fac980a6000 comp rx=0 tx=0).stop 2024-04-18T10:06:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7faca9bfa640 1 -- 172.21.15.62:0/654200787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faca409c4d0 msgr2=0x7faca40b7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7faca9bfa640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faca409c4d0 0x7faca40b7cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fac9400e9d0 tx=0x7fac9400ee90 comp rx=0 tx=0).stop 2024-04-18T10:06:35.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7faca9bfa640 1 -- 172.21.15.62:0/654200787 shutdown_connections 2024-04-18T10:06:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7faca9bfa640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fac90068520 0x7fac9006a9e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7faca9bfa640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faca41497f0 0x7faca414e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7faca9bfa640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faca409c4d0 0x7faca40b7cc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7faca9bfa640 1 --2- 172.21.15.62:0/654200787 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faca409b620 0x7faca413ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7faca9bfa640 1 -- 172.21.15.62:0/654200787 >> 172.21.15.62:0/654200787 conn(0x7faca40b9420 msgr2=0x7faca406d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7faca9bfa640 1 -- 172.21.15.62:0/654200787 shutdown_connections 2024-04-18T10:06:35.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.521+0000 7faca9bfa640 1 -- 172.21.15.62:0/654200787 wait complete. 2024-04-18T10:06:35.537 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:35.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.657+0000 7f73b0260640 1 Processor -- start 2024-04-18T10:06:35.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.657+0000 7f73b0260640 1 -- start start 2024-04-18T10:06:35.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73b0260640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f73a8145f20 0x7f73a81514f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73b0260640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a806e030 con 0x7f73a809be30 2024-04-18T10:06:35.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73b0260640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73a80ba170 con 0x7f73a80b9d90 2024-04-18T10:06:35.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73b0260640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f73a809cd30 con 0x7f73a8145f20 2024-04-18T10:06:35.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73ad7fb640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f73a809be30 0x7f73a806d920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33836/0 (socket says 172.21.15.62:33836) 2024-04-18T10:06:35.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73ad7fb640 1 -- 172.21.15.62:0/1004050622 learned_addr learned my addr 172.21.15.62:0/1004050622 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:35.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73ae7fd640 1 --2- 172.21.15.62:0/1004050622 >> v2:172.21.15.145:3300/0 conn(0x7f73a8145f20 0x7f73a81514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73acffa640 1 -- 172.21.15.62:0/1004050622 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2708984320 0 0) 0x7f73a806e030 con 0x7f73a809be30 2024-04-18T10:06:35.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73acffa640 1 -- 172.21.15.62:0/1004050622 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7394003530 con 0x7f73a809be30 2024-04-18T10:06:35.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73ae7fd640 1 -- 172.21.15.62:0/1004050622 >> v1:172.21.15.145:6789/0 conn(0x7f73a80b9d90 legacy=0x7f73a806d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73ae7fd640 1 -- 172.21.15.62:0/1004050622 >> v1:172.21.15.62:6789/0 conn(0x7f73a809be30 legacy=0x7f73a806d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73acffa640 1 -- 172.21.15.62:0/1004050622 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2210946866 0 0) 0x7f7394003530 con 0x7f73a809be30 2024-04-18T10:06:35.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73ae7fd640 1 -- 172.21.15.62:0/1004050622 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a814a060 con 0x7f73a8145f20 2024-04-18T10:06:35.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73ae7fd640 1 --2- 172.21.15.62:0/1004050622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73a8145f20 0x7f73a81514f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f73a0009d90 tx=0x7f73a001c770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d27eddedc45459d server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:35.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73acffa640 1 -- 172.21.15.62:0/1004050622 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7398001680 con 0x7f73a8145f20 2024-04-18T10:06:35.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73acffa640 1 -- 172.21.15.62:0/1004050622 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73a00a64b0 con 0x7f73a8145f20 2024-04-18T10:06:35.663 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.661+0000 7f73acffa640 1 -- 172.21.15.62:0/1004050622 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73a00a6860 con 0x7f73a8145f20 2024-04-18T10:06:35.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 -- 172.21.15.62:0/1004050622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73a8145f20 msgr2=0x7f73a81514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 --2- 172.21.15.62:0/1004050622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73a8145f20 0x7f73a81514f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f73a0009d90 tx=0x7f73a001c770 comp rx=0 tx=0).stop 2024-04-18T10:06:35.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 -- 172.21.15.62:0/1004050622 shutdown_connections 2024-04-18T10:06:35.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 --2- 172.21.15.62:0/1004050622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73a8145f20 0x7f73a81514f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 -- 172.21.15.62:0/1004050622 >> 172.21.15.62:0/1004050622 conn(0x7f73a806c0d0 msgr2=0x7f73a80b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:35.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 -- 172.21.15.62:0/1004050622 shutdown_connections 2024-04-18T10:06:35.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 -- 172.21.15.62:0/1004050622 wait complete. 2024-04-18T10:06:35.664 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 Processor -- start 2024-04-18T10:06:35.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 -- start start 2024-04-18T10:06:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73a809be30 0x7f73a8071b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a80b9d90 0x7f73a8099410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73a8145f20 0x7f73a8099950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f73a80b3e20 con 0x7f73a80b9d90 2024-04-18T10:06:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f73a8146390 con 0x7f73a8145f20 2024-04-18T10:06:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f73a80b3fa0 con 0x7f73a809be30 2024-04-18T10:06:35.667 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73ad7fb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a80b9d90 0x7f73a8099410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73ad7fb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a80b9d90 0x7f73a8099410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41014/0 (socket says 172.21.15.62:41014) 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73ad7fb640 1 -- 172.21.15.62:0/4205646816 learned_addr learned my addr 172.21.15.62:0/4205646816 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73ae7fd640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73a8145f20 0x7f73a8099950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73adffc640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73a809be30 0x7f73a8071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73ad7fb640 1 -- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73a809be30 msgr2=0x7f73a8071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73ad7fb640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73a809be30 0x7f73a8071b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73ad7fb640 1 -- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73a8145f20 msgr2=0x7f73a8099950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73ad7fb640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73a8145f20 0x7f73a8099950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73ad7fb640 1 -- 172.21.15.62:0/4205646816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73a8073180 con 0x7f73a80b9d90 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73ad7fb640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a80b9d90 0x7f73a8099410 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f739800c760 tx=0x7f739800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f739e7fc640 1 -- 172.21.15.62:0/4205646816 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7398017070 con 0x7f73a80b9d90 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f739e7fc640 1 -- 172.21.15.62:0/4205646816 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7398005b00 con 0x7f73a80b9d90 2024-04-18T10:06:35.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f739e7fc640 1 -- 172.21.15.62:0/4205646816 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7398007070 con 0x7f73a80b9d90 2024-04-18T10:06:35.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 -- 172.21.15.62:0/4205646816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73a8070e90 con 0x7f73a80b9d90 2024-04-18T10:06:35.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 -- 172.21.15.62:0/4205646816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73a8070a90 con 0x7f73a80b9d90 2024-04-18T10:06:35.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.665+0000 7f73b0260640 1 -- 172.21.15.62:0/4205646816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73a8068b90 con 0x7f73a80b9d90 2024-04-18T10:06:35.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.669+0000 7f739e7fc640 1 -- 172.21.15.62:0/4205646816 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f739801a070 con 0x7f73a80b9d90 2024-04-18T10:06:35.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.669+0000 7f739e7fc640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f738c0683b0 0x7f738c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.669+0000 7f739e7fc640 1 -- 172.21.15.62:0/4205646816 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f7398085b00 con 0x7f73a80b9d90 2024-04-18T10:06:35.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.669+0000 7f73adffc640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f738c0683b0 0x7f738c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.669+0000 7f73adffc640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f738c0683b0 0x7f738c06a870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f73a80b38e0 tx=0x7f73a4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:35.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.673+0000 7f739e7fc640 1 -- 172.21.15.62:0/4205646816 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f73a8070e90 con 0x7f73a80b9d90 2024-04-18T10:06:35.836 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.837+0000 7f73b0260640 1 -- 172.21.15.62:0/4205646816 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73a806cae0 con 0x7f73a80b9d90 2024-04-18T10:06:35.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.837+0000 7f739e7fc640 1 -- 172.21.15.62:0/4205646816 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+12879 (secure 0 0 0) 0x7f73980564a0 con 0x7f73a80b9d90 2024-04-18T10:06:35.839 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:35.839 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":26,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:32.900090+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:35.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.841+0000 7f73b0260640 1 -- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f738c0683b0 msgr2=0x7f738c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.841+0000 7f73b0260640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f738c0683b0 0x7f738c06a870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f73a80b38e0 tx=0x7f73a4009040 comp rx=0 tx=0).stop 2024-04-18T10:06:35.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.841+0000 7f73b0260640 1 -- 172.21.15.62:0/4205646816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a80b9d90 msgr2=0x7f73a8099410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.841+0000 7f73b0260640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a80b9d90 0x7f73a8099410 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f739800c760 tx=0x7f739800cc20 comp rx=0 tx=0).stop 2024-04-18T10:06:35.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.841+0000 7f73b0260640 1 -- 172.21.15.62:0/4205646816 shutdown_connections 2024-04-18T10:06:35.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.841+0000 7f73b0260640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f738c0683b0 0x7f738c06a870 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.841+0000 7f73b0260640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73a8145f20 0x7f73a8099950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.841+0000 7f73b0260640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f73a80b9d90 0x7f73a8099410 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.841+0000 7f73b0260640 1 --2- 172.21.15.62:0/4205646816 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73a809be30 0x7f73a8071b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.841+0000 7f73b0260640 1 -- 172.21.15.62:0/4205646816 >> 172.21.15.62:0/4205646816 conn(0x7f73a806c0d0 msgr2=0x7f73a809d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:35.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.841+0000 7f73b0260640 1 -- 172.21.15.62:0/4205646816 shutdown_connections 2024-04-18T10:06:35.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.841+0000 7f73b0260640 1 -- 172.21.15.62:0/4205646816 wait complete. 2024-04-18T10:06:35.859 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:35.859 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:35.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b9dfc640 1 Processor -- start 2024-04-18T10:06:35.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b9dfc640 1 -- start start 2024-04-18T10:06:35.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b9dfc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f53b40b3eb0 0x7f53b40bdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b9dfc640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b414b620 con 0x7f53b409b9d0 2024-04-18T10:06:35.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b9dfc640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53b414b900 con 0x7f53b414b190 2024-04-18T10:06:35.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b9dfc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f53b40b4290 con 0x7f53b40b3eb0 2024-04-18T10:06:35.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b3fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f53b409b9d0 0x7f53b4152520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33850/0 (socket says 172.21.15.62:33850) 2024-04-18T10:06:35.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b3fff640 1 -- 172.21.15.62:0/1886278552 learned_addr learned my addr 172.21.15.62:0/1886278552 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:35.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b37fe640 1 --2- 172.21.15.62:0/1886278552 >> v2:172.21.15.145:3300/0 conn(0x7f53b40b3eb0 0x7f53b40bdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3622940299 0 0) 0x7f53b414b620 con 0x7f53b409b9d0 2024-04-18T10:06:35.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f539c003530 con 0x7f53b409b9d0 2024-04-18T10:06:35.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 608157919 0 0) 0x7f539c003530 con 0x7f53b409b9d0 2024-04-18T10:06:35.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 >> v2:172.21.15.145:3300/0 conn(0x7f53b40b3eb0 msgr2=0x7f53b40bdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:35.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 --2- 172.21.15.62:0/1886278552 >> v2:172.21.15.145:3300/0 conn(0x7f53b40b3eb0 0x7f53b40bdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 >> v1:172.21.15.145:6789/0 conn(0x7f53b414b190 legacy=0x7f53b4147ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b40be270 con 0x7f53b409b9d0 2024-04-18T10:06:35.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f53ac001a00 con 0x7f53b409b9d0 2024-04-18T10:06:35.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 >> v1:172.21.15.62:6789/0 conn(0x7f53b409b9d0 legacy=0x7f53b4152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:35.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 --2- 172.21.15.62:0/1886278552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 0x7f539c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 --2- 172.21.15.62:0/1886278552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f539c004810 0x7f539c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 --2- 172.21.15.62:0/1886278552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f539c005220 0x7f539c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:35.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f53b40b4290 con 0x7f539c003a10 2024-04-18T10:06:35.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f539c00c170 con 0x7f539c004810 2024-04-18T10:06:35.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f539c00c370 con 0x7f539c005220 2024-04-18T10:06:35.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b37fe640 1 --2- 172.21.15.62:0/1886278552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 0x7f539c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b2ffd640 1 --2- 172.21.15.62:0/1886278552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f539c004810 0x7f539c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b37fe640 1 -- 172.21.15.62:0/1886278552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f539c005220 msgr2=0x7f539c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:35.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b3fff640 1 --2- 172.21.15.62:0/1886278552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f539c005220 0x7f539c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:35.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b37fe640 1 --2- 172.21.15.62:0/1886278552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f539c005220 0x7f539c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b37fe640 1 -- 172.21.15.62:0/1886278552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f539c004810 msgr2=0x7f539c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:35.998 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b37fe640 1 --2- 172.21.15.62:0/1886278552 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f539c004810 0x7f539c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:35.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b37fe640 1 -- 172.21.15.62:0/1886278552 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f539c003530 con 0x7f539c003a10 2024-04-18T10:06:35.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b3fff640 1 --2- 172.21.15.62:0/1886278552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f539c005220 0x7f539c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:35.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.993+0000 7f53b3fff640 1 -- 172.21.15.62:0/1886278552 reap_dead start 2024-04-18T10:06:36.002 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b37fe640 1 --2- 172.21.15.62:0/1886278552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 0x7f539c003e30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009a90 tx=0x7f53a401ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba46e4501f55fd09 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:36.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4009690 con 0x7f539c003a10 2024-04-18T10:06:36.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a40a8440 con 0x7f539c003a10 2024-04-18T10:06:36.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b27fc640 1 -- 172.21.15.62:0/1886278552 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a40a8810 con 0x7f539c003a10 2024-04-18T10:06:36.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b9dfc640 1 -- 172.21.15.62:0/1886278552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 msgr2=0x7f539c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b9dfc640 1 --2- 172.21.15.62:0/1886278552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 0x7f539c003e30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009a90 tx=0x7f53a401ca00 comp rx=0 tx=0).stop 2024-04-18T10:06:36.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b9dfc640 1 -- 172.21.15.62:0/1886278552 shutdown_connections 2024-04-18T10:06:36.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b9dfc640 1 --2- 172.21.15.62:0/1886278552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 0x7f539c003e30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b9dfc640 1 -- 172.21.15.62:0/1886278552 >> 172.21.15.62:0/1886278552 conn(0x7f53b4067e10 msgr2=0x7f53b40bd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:36.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b9dfc640 1 -- 172.21.15.62:0/1886278552 shutdown_connections 2024-04-18T10:06:36.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b9dfc640 1 -- 172.21.15.62:0/1886278552 wait complete. 2024-04-18T10:06:36.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b9dfc640 1 Processor -- start 2024-04-18T10:06:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:35.997+0000 7f53b9dfc640 1 -- start start 2024-04-18T10:06:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b9dfc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 0x7f53b40b8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b9dfc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53b4142510 0x7f53b41e0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b9dfc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53b41e0d40 0x7f53b41e3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b9dfc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f53b4147f30 con 0x7f539c003a10 2024-04-18T10:06:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b9dfc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f53b4147db0 con 0x7f53b41e0d40 2024-04-18T10:06:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b9dfc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f53b41480b0 con 0x7f53b4142510 2024-04-18T10:06:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 0x7f53b40b8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 0x7f53b40b8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41018/0 (socket says 172.21.15.62:41018) 2024-04-18T10:06:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b37fe640 1 -- 172.21.15.62:0/3269178991 learned_addr learned my addr 172.21.15.62:0/3269178991 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b37fe640 1 -- 172.21.15.62:0/3269178991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53b4142510 msgr2=0x7f53b41e0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b37fe640 1 --2- 172.21.15.62:0/3269178991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53b4142510 0x7f53b41e0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b37fe640 1 -- 172.21.15.62:0/3269178991 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53b41e0d40 msgr2=0x7f53b41e3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b37fe640 1 --2- 172.21.15.62:0/3269178991 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53b41e0d40 0x7f53b41e3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b37fe640 1 -- 172.21.15.62:0/3269178991 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b409e870 con 0x7f539c003a10 2024-04-18T10:06:36.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b37fe640 1 --2- 172.21.15.62:0/3269178991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 0x7f53b40b8e00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009a90 tx=0x7f53a40197a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:36.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f539bfff640 1 -- 172.21.15.62:0/3269178991 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4009690 con 0x7f539c003a10 2024-04-18T10:06:36.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f539bfff640 1 -- 172.21.15.62:0/3269178991 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a40bb640 con 0x7f539c003a10 2024-04-18T10:06:36.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b9dfc640 1 -- 172.21.15.62:0/3269178991 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b4143820 con 0x7f539c003a10 2024-04-18T10:06:36.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f539bfff640 1 -- 172.21.15.62:0/3269178991 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a40bba30 con 0x7f539c003a10 2024-04-18T10:06:36.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b9dfc640 1 -- 172.21.15.62:0/3269178991 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b4143540 con 0x7f539c003a10 2024-04-18T10:06:36.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b9dfc640 1 -- 172.21.15.62:0/3269178991 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53b4068c00 con 0x7f539c003a10 2024-04-18T10:06:36.005 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f539bfff640 1 -- 172.21.15.62:0/3269178991 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f53a40bbbd0 con 0x7f539c003a10 2024-04-18T10:06:36.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f539bfff640 1 --2- 172.21.15.62:0/3269178991 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f53a00683b0 0x7f53a006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f539bfff640 1 -- 172.21.15.62:0/3269178991 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f53a41210e0 con 0x7f539c003a10 2024-04-18T10:06:36.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.001+0000 7f53b2ffd640 1 --2- 172.21.15.62:0/3269178991 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f53a00683b0 0x7f53a006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.005+0000 7f53b2ffd640 1 --2- 172.21.15.62:0/3269178991 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f53a00683b0 0x7f53a006a870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f53a8007ab0 tx=0x7f53a8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:36.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.005+0000 7f539bfff640 1 -- 172.21.15.62:0/3269178991 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f53a40b7070 con 0x7f539c003a10 2024-04-18T10:06:36.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.177+0000 7f53b9dfc640 1 -- 172.21.15.62:0/3269178991 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f53b409fda0 con 0x7f539c003a10 2024-04-18T10:06:36.179 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f539bfff640 1 -- 172.21.15.62:0/3269178991 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f53a40ac030 con 0x7f539c003a10 2024-04-18T10:06:36.179 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:36.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f53b9dfc640 1 -- 172.21.15.62:0/3269178991 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f53a00683b0 msgr2=0x7f53a006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f53b9dfc640 1 --2- 172.21.15.62:0/3269178991 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f53a00683b0 0x7f53a006a870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f53a8007ab0 tx=0x7f53a8009040 comp rx=0 tx=0).stop 2024-04-18T10:06:36.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f53b9dfc640 1 -- 172.21.15.62:0/3269178991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 msgr2=0x7f53b40b8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f53b9dfc640 1 --2- 172.21.15.62:0/3269178991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 0x7f53b40b8e00 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009a90 tx=0x7f53a40197a0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f53b9dfc640 1 -- 172.21.15.62:0/3269178991 shutdown_connections 2024-04-18T10:06:36.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f53b9dfc640 1 --2- 172.21.15.62:0/3269178991 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f53a00683b0 0x7f53a006a870 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f53b9dfc640 1 --2- 172.21.15.62:0/3269178991 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53b41e0d40 0x7f53b41e3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f53b9dfc640 1 --2- 172.21.15.62:0/3269178991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53b4142510 0x7f53b41e0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f53b9dfc640 1 --2- 172.21.15.62:0/3269178991 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f539c003a10 0x7f53b40b8e00 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f53b9dfc640 1 -- 172.21.15.62:0/3269178991 >> 172.21.15.62:0/3269178991 conn(0x7f53b4067e10 msgr2=0x7f53b40ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:36.182 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f53b9dfc640 1 -- 172.21.15.62:0/3269178991 shutdown_connections 2024-04-18T10:06:36.183 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.181+0000 7f53b9dfc640 1 -- 172.21.15.62:0/3269178991 wait complete. 2024-04-18T10:06:36.197 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-18T10:06:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca3817640 1 Processor -- start 2024-04-18T10:06:36.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca3817640 1 -- start start 2024-04-18T10:06:36.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca3817640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2c9c067810 0x7f2c9c06da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca3817640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c9c145c50 con 0x7f2c9c146730 2024-04-18T10:06:36.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca3817640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c9c145f30 con 0x7f2c9c067bf0 2024-04-18T10:06:36.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca3817640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f2c9c1401c0 con 0x7f2c9c067810 2024-04-18T10:06:36.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca0db2640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f2c9c146730 0x7f2c9c0b5f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33856/0 (socket says 172.21.15.62:33856) 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca0db2640 1 -- 172.21.15.62:0/3241619913 learned_addr learned my addr 172.21.15.62:0/3241619913 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca15b3640 1 --2- 172.21.15.62:0/3241619913 >> v2:172.21.15.145:3300/0 conn(0x7f2c9c067810 0x7f2c9c06da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4274278542 0 0) 0x7f2c9c145c50 con 0x7f2c9c146730 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2c88003530 con 0x7f2c9c146730 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 152651761 0 0) 0x7f2c88003530 con 0x7f2c9c146730 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 >> v2:172.21.15.145:3300/0 conn(0x7f2c9c067810 msgr2=0x7f2c9c06da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 --2- 172.21.15.62:0/3241619913 >> v2:172.21.15.145:3300/0 conn(0x7f2c9c067810 0x7f2c9c06da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 >> v1:172.21.15.145:6789/0 conn(0x7f2c9c067bf0 legacy=0x7f2c9c13fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c9c146210 con 0x7f2c9c146730 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f2c98001800 con 0x7f2c9c146730 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 >> v1:172.21.15.62:6789/0 conn(0x7f2c9c146730 legacy=0x7f2c9c0b5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 --2- 172.21.15.62:0/3241619913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c880039c0 0x7f2c88003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 --2- 172.21.15.62:0/3241619913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c88004810 0x7f2c88004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 --2- 172.21.15.62:0/3241619913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c880051d0 0x7f2c8800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8800c140 con 0x7f2c88004810 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8800c2f0 con 0x7f2c880051d0 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2c8800c4f0 con 0x7f2c880039c0 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca0db2640 1 --2- 172.21.15.62:0/3241619913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c88004810 0x7f2c88004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca1db4640 1 --2- 172.21.15.62:0/3241619913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c880051d0 0x7f2c8800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca15b3640 1 --2- 172.21.15.62:0/3241619913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c880039c0 0x7f2c88003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca0db2640 1 -- 172.21.15.62:0/3241619913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c880039c0 msgr2=0x7f2c88003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca0db2640 1 --2- 172.21.15.62:0/3241619913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c880039c0 0x7f2c88003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca0db2640 1 -- 172.21.15.62:0/3241619913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c880051d0 msgr2=0x7f2c8800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca0db2640 1 --2- 172.21.15.62:0/3241619913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c880051d0 0x7f2c8800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca0db2640 1 -- 172.21.15.62:0/3241619913 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c88003530 con 0x7f2c88004810 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca1db4640 1 -- 172.21.15.62:0/3241619913 reap_dead start 2024-04-18T10:06:36.313 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.309+0000 7f2ca15b3640 1 --2- 172.21.15.62:0/3241619913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c880039c0 0x7f2c88003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca0db2640 1 --2- 172.21.15.62:0/3241619913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c88004810 0x7f2c88004c90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2c9800c210 tx=0x7f2c9801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648a1e60bd79d8c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98007d10 con 0x7f2c88004810 2024-04-18T10:06:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c980a9070 con 0x7f2c88004810 2024-04-18T10:06:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2c93fff640 1 -- 172.21.15.62:0/3241619913 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98007510 con 0x7f2c88004810 2024-04-18T10:06:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 -- 172.21.15.62:0/3241619913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c88004810 msgr2=0x7f2c88004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 --2- 172.21.15.62:0/3241619913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c88004810 0x7f2c88004c90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2c9800c210 tx=0x7f2c9801cc60 comp rx=0 tx=0).stop 2024-04-18T10:06:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 -- 172.21.15.62:0/3241619913 shutdown_connections 2024-04-18T10:06:36.316 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 --2- 172.21.15.62:0/3241619913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c88004810 0x7f2c88004c90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 -- 172.21.15.62:0/3241619913 >> 172.21.15.62:0/3241619913 conn(0x7f2c9c06c3c0 msgr2=0x7f2c9c0b9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 -- 172.21.15.62:0/3241619913 shutdown_connections 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 -- 172.21.15.62:0/3241619913 wait complete. 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 Processor -- start 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 -- start start 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c88004810 0x7f2c9c142700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c9c1dee20 0x7f2c9c13d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c9c140fa0 0x7f2c9c141420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9c149440 con 0x7f2c9c140fa0 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f2c9c1492c0 con 0x7f2c88004810 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca3817640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2c9c1495c0 con 0x7f2c9c1dee20 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca1db4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c9c140fa0 0x7f2c9c141420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca1db4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c9c140fa0 0x7f2c9c141420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41032/0 (socket says 172.21.15.62:41032) 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca1db4640 1 -- 172.21.15.62:0/1661678378 learned_addr learned my addr 172.21.15.62:0/1661678378 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca0db2640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c9c1dee20 0x7f2c9c13d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.317 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca15b3640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c88004810 0x7f2c9c142700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca1db4640 1 -- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c9c1dee20 msgr2=0x7f2c9c13d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca1db4640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c9c1dee20 0x7f2c9c13d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca1db4640 1 -- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c88004810 msgr2=0x7f2c9c142700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca1db4640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c88004810 0x7f2c9c142700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca1db4640 1 -- 172.21.15.62:0/1661678378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c9c141960 con 0x7f2c9c140fa0 2024-04-18T10:06:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.313+0000 7f2ca1db4640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c9c140fa0 0x7f2c9c141420 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c068040 tx=0x7f2c9400e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.317+0000 7f2c91ffb640 1 -- 172.21.15.62:0/1661678378 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c94019070 con 0x7f2c9c140fa0 2024-04-18T10:06:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.317+0000 7f2c91ffb640 1 -- 172.21.15.62:0/1661678378 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c94006e30 con 0x7f2c9c140fa0 2024-04-18T10:06:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.317+0000 7f2ca3817640 1 -- 172.21.15.62:0/1661678378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c9c1df630 con 0x7f2c9c140fa0 2024-04-18T10:06:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.317+0000 7f2c91ffb640 1 -- 172.21.15.62:0/1661678378 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c94003b80 con 0x7f2c9c140fa0 2024-04-18T10:06:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.317+0000 7f2ca3817640 1 -- 172.21.15.62:0/1661678378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c9c1df2a0 con 0x7f2c9c140fa0 2024-04-18T10:06:36.318 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.317+0000 7f2ca3817640 1 -- 172.21.15.62:0/1661678378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c9c142c80 con 0x7f2c9c140fa0 2024-04-18T10:06:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.317+0000 7f2c91ffb640 1 -- 172.21.15.62:0/1661678378 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2c9401c070 con 0x7f2c9c140fa0 2024-04-18T10:06:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.317+0000 7f2c91ffb640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2c80068380 0x7f2c8006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.317+0000 7f2c91ffb640 1 -- 172.21.15.62:0/1661678378 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f2c94085b10 con 0x7f2c9c140fa0 2024-04-18T10:06:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.317+0000 7f2ca15b3640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2c80068380 0x7f2c8006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.319 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.317+0000 7f2ca15b3640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2c80068380 0x7f2c8006a840 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c140b10 tx=0x7f2c8c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:36.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.321+0000 7f2c91ffb640 1 -- 172.21.15.62:0/1661678378 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2c9c141960 con 0x7f2c9c140fa0 2024-04-18T10:06:36.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.481+0000 7f2ca3817640 1 -- 172.21.15.62:0/1661678378 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f2c9c069230 con 0x7f2c9c140fa0 2024-04-18T10:06:36.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.481+0000 7f2c91ffb640 1 -- 172.21.15.62:0/1661678378 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v26) ==== 84+0+65 (secure 0 0 0) 0x7f2c94055e60 con 0x7f2c9c140fa0 2024-04-18T10:06:36.482 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:36.482 INFO:teuthology.orchestra.run.smithi062.stdout:[ 2024-04-18T10:06:36.482 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-04-18T10:06:36.482 INFO:teuthology.orchestra.run.smithi062.stdout: "poolnum": 1, 2024-04-18T10:06:36.482 INFO:teuthology.orchestra.run.smithi062.stdout: "poolname": ".mgr" 2024-04-18T10:06:36.482 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:36.482 INFO:teuthology.orchestra.run.smithi062.stdout:] 2024-04-18T10:06:36.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.485+0000 7f2ca3817640 1 -- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2c80068380 msgr2=0x7f2c8006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.485+0000 7f2ca3817640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2c80068380 0x7f2c8006a840 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c140b10 tx=0x7f2c8c008040 comp rx=0 tx=0).stop 2024-04-18T10:06:36.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.485+0000 7f2ca3817640 1 -- 172.21.15.62:0/1661678378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c9c140fa0 msgr2=0x7f2c9c141420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.485+0000 7f2ca3817640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c9c140fa0 0x7f2c9c141420 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c068040 tx=0x7f2c9400e9e0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.485+0000 7f2ca3817640 1 -- 172.21.15.62:0/1661678378 shutdown_connections 2024-04-18T10:06:36.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.485+0000 7f2ca3817640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2c80068380 0x7f2c8006a840 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.485+0000 7f2ca3817640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c9c140fa0 0x7f2c9c141420 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.485+0000 7f2ca3817640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c9c1dee20 0x7f2c9c13d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.485+0000 7f2ca3817640 1 --2- 172.21.15.62:0/1661678378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c88004810 0x7f2c9c142700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.485+0000 7f2ca3817640 1 -- 172.21.15.62:0/1661678378 >> 172.21.15.62:0/1661678378 conn(0x7f2c9c06c3c0 msgr2=0x7f2c9c1447c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.485+0000 7f2ca3817640 1 -- 172.21.15.62:0/1661678378 shutdown_connections 2024-04-18T10:06:36.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.485+0000 7f2ca3817640 1 -- 172.21.15.62:0/1661678378 wait complete. 2024-04-18T10:06:36.499 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-18T10:06:36.500 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-18T10:06:36.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f464c7d5640 1 Processor -- start 2024-04-18T10:06:36.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f464c7d5640 1 -- start start 2024-04-18T10:06:36.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f464c7d5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46440b3e20 0x7f46441461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f464c7d5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f464406d920 0x7f4644150470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f464c7d5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46440ba170 con 0x7f46440b9d90 2024-04-18T10:06:36.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f464c7d5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f464409cca0 con 0x7f464406d920 2024-04-18T10:06:36.617 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f464c7d5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f4644149930 con 0x7f46440b3e20 2024-04-18T10:06:36.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f4649d70640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46440b3e20 0x7f46441461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f464ad72640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f464406d920 0x7f4644150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f4649d70640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46440b3e20 0x7f46441461e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:59738/0 (socket says 172.21.15.62:59738) 2024-04-18T10:06:36.618 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f4649d70640 1 -- 172.21.15.62:0/1256618652 learned_addr learned my addr 172.21.15.62:0/1256618652 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:36.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f464ad72640 1 -- 172.21.15.62:0/1256618652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46440b3e20 msgr2=0x7f46441461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:36.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f464ad72640 1 --2- 172.21.15.62:0/1256618652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46440b3e20 0x7f46441461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f464ad72640 1 -- 172.21.15.62:0/1256618652 >> v1:172.21.15.145:6789/0 conn(0x7f46440b9d90 legacy=0x7f464406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.619 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.617+0000 7f464ad72640 1 -- 172.21.15.62:0/1256618652 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4644149c10 con 0x7f464406d920 2024-04-18T10:06:36.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464ad72640 1 --2- 172.21.15.62:0/1256618652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f464406d920 0x7f4644150470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f463c0097c0 tx=0x7f463c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e818874724542fbb server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:36.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464956f640 1 -- 172.21.15.62:0/1256618652 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c0b0070 con 0x7f464406d920 2024-04-18T10:06:36.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464956f640 1 -- 172.21.15.62:0/1256618652 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f463c0ab600 con 0x7f464406d920 2024-04-18T10:06:36.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464956f640 1 -- 172.21.15.62:0/1256618652 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c0ab9d0 con 0x7f464406d920 2024-04-18T10:06:36.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 -- 172.21.15.62:0/1256618652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f464406d920 msgr2=0x7f4644150470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.621 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 --2- 172.21.15.62:0/1256618652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f464406d920 0x7f4644150470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f463c0097c0 tx=0x7f463c01c410 comp rx=0 tx=0).stop 2024-04-18T10:06:36.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 -- 172.21.15.62:0/1256618652 shutdown_connections 2024-04-18T10:06:36.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 --2- 172.21.15.62:0/1256618652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f464406d920 0x7f4644150470 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 --2- 172.21.15.62:0/1256618652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46440b3e20 0x7f46441461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 -- 172.21.15.62:0/1256618652 >> 172.21.15.62:0/1256618652 conn(0x7f464406c0d0 msgr2=0x7f46440b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:36.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 -- 172.21.15.62:0/1256618652 shutdown_connections 2024-04-18T10:06:36.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 -- 172.21.15.62:0/1256618652 wait complete. 2024-04-18T10:06:36.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 Processor -- start 2024-04-18T10:06:36.622 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 -- start start 2024-04-18T10:06:36.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464406d920 0x7f46440b1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46440b3e20 0x7f46440b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46440b9d90 0x7f46441e0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.621+0000 7f464c7d5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f46441533e0 con 0x7f464406d920 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464c7d5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f464409c020 con 0x7f46440b3e20 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464c7d5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f4644153560 con 0x7f46440b9d90 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464a571640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464406d920 0x7f46440b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464a571640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464406d920 0x7f46440b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41044/0 (socket says 172.21.15.62:41044) 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464a571640 1 -- 172.21.15.62:0/2381752195 learned_addr learned my addr 172.21.15.62:0/2381752195 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464ad72640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46440b9d90 0x7f46441e0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f4649d70640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46440b3e20 0x7f46440b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464a571640 1 -- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46440b9d90 msgr2=0x7f46441e0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464a571640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46440b9d90 0x7f46441e0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464a571640 1 -- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46440b3e20 msgr2=0x7f46440b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464a571640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46440b3e20 0x7f46440b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464a571640 1 -- 172.21.15.62:0/2381752195 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4644070750 con 0x7f464406d920 2024-04-18T10:06:36.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464a571640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464406d920 0x7f46440b1260 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f464000c790 tx=0x7f464000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:36.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f463affd640 1 -- 172.21.15.62:0/2381752195 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640016020 con 0x7f464406d920 2024-04-18T10:06:36.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f463affd640 1 -- 172.21.15.62:0/2381752195 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4640006d60 con 0x7f464406d920 2024-04-18T10:06:36.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464c7d5640 1 -- 172.21.15.62:0/2381752195 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4644070a30 con 0x7f464406d920 2024-04-18T10:06:36.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f463affd640 1 -- 172.21.15.62:0/2381752195 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640003bc0 con 0x7f464406d920 2024-04-18T10:06:36.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464c7d5640 1 -- 172.21.15.62:0/2381752195 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46440b2660 con 0x7f464406d920 2024-04-18T10:06:36.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f464c7d5640 1 -- 172.21.15.62:0/2381752195 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4644068ca0 con 0x7f464406d920 2024-04-18T10:06:36.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f463affd640 1 -- 172.21.15.62:0/2381752195 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f464000a850 con 0x7f464406d920 2024-04-18T10:06:36.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f463affd640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4628068380 0x7f462806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:36.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f463affd640 1 -- 172.21.15.62:0/2381752195 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 4857+0+0 (secure 0 0 0) 0x7f46400871b0 con 0x7f464406d920 2024-04-18T10:06:36.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.625+0000 7f4649d70640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4628068380 0x7f462806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:36.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.629+0000 7f4649d70640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4628068380 0x7f462806a840 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f46440b28c0 tx=0x7f4634008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:36.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.629+0000 7f463affd640 1 -- 172.21.15.62:0/2381752195 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4640020020 con 0x7f464406d920 2024-04-18T10:06:36.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.793+0000 7f464c7d5640 1 -- 172.21.15.62:0/2381752195 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f46440af3a0 con 0x7f464406d920 2024-04-18T10:06:36.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.921+0000 7f463affd640 1 -- 172.21.15.62:0/2381752195 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v27) ==== 138+0+0 (secure 0 0 0) 0x7f4640057b30 con 0x7f464406d920 2024-04-18T10:06:36.922 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' created 2024-04-18T10:06:36.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.925+0000 7f464c7d5640 1 -- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4628068380 msgr2=0x7f462806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.925+0000 7f464c7d5640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4628068380 0x7f462806a840 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f46440b28c0 tx=0x7f4634008040 comp rx=0 tx=0).stop 2024-04-18T10:06:36.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.925+0000 7f464c7d5640 1 -- 172.21.15.62:0/2381752195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464406d920 msgr2=0x7f46440b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:36.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.925+0000 7f464c7d5640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464406d920 0x7f46440b1260 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f464000c790 tx=0x7f464000cc50 comp rx=0 tx=0).stop 2024-04-18T10:06:36.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.925+0000 7f464c7d5640 1 -- 172.21.15.62:0/2381752195 shutdown_connections 2024-04-18T10:06:36.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.925+0000 7f464c7d5640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f4628068380 0x7f462806a840 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.925+0000 7f464c7d5640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46440b9d90 0x7f46441e0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.925+0000 7f464c7d5640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46440b3e20 0x7f46440b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.925+0000 7f464c7d5640 1 --2- 172.21.15.62:0/2381752195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f464406d920 0x7f46440b1260 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:36.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.925+0000 7f464c7d5640 1 -- 172.21.15.62:0/2381752195 >> 172.21.15.62:0/2381752195 conn(0x7f464406c0d0 msgr2=0x7f46440b6670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:36.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.925+0000 7f464c7d5640 1 -- 172.21.15.62:0/2381752195 shutdown_connections 2024-04-18T10:06:36.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:36.925+0000 7f464c7d5640 1 -- 172.21.15.62:0/2381752195 wait complete. 2024-04-18T10:06:36.952 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-18T10:06:37.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c662a5640 1 Processor -- start 2024-04-18T10:06:37.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c662a5640 1 -- start start 2024-04-18T10:06:37.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c662a5640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7c6006d210 0x7f7c60145a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c662a5640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c600b30e0 con 0x7f7c60073cd0 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c662a5640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c6006e580 con 0x7f7c600b2d00 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c662a5640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f7c600b6d00 con 0x7f7c6006d210 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c64842640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7c60073cd0 0x7f7c60150390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33870/0 (socket says 172.21.15.62:33870) 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c64842640 1 -- 172.21.15.62:0/937346594 learned_addr learned my addr 172.21.15.62:0/937346594 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5f7fe640 1 --2- 172.21.15.62:0/937346594 >> v2:172.21.15.145:3300/0 conn(0x7f7c6006d210 0x7f7c60145a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1430732637 0 0) 0x7f7c600b30e0 con 0x7f7c60073cd0 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c48003530 con 0x7f7c60073cd0 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1730607434 0 0) 0x7f7c48003530 con 0x7f7c60073cd0 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 >> v2:172.21.15.145:3300/0 conn(0x7f7c6006d210 msgr2=0x7f7c60145a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 --2- 172.21.15.62:0/937346594 >> v2:172.21.15.145:3300/0 conn(0x7f7c6006d210 0x7f7c60145a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 >> v1:172.21.15.145:6789/0 conn(0x7f7c600b2d00 legacy=0x7f7c600b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:37.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c6006e000 con 0x7f7c60073cd0 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f7c50001680 con 0x7f7c60073cd0 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 >> v1:172.21.15.62:6789/0 conn(0x7f7c60073cd0 legacy=0x7f7c60150390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 --2- 172.21.15.62:0/937346594 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c480039c0 0x7f7c48003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 --2- 172.21.15.62:0/937346594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 0x7f7c48004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 --2- 172.21.15.62:0/937346594 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c480052f0 0x7f7c4800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7c600b6d00 con 0x7f7c48004810 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4800c1c0 con 0x7f7c480052f0 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4800c3c0 con 0x7f7c480039c0 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5f7fe640 1 --2- 172.21.15.62:0/937346594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 0x7f7c48004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5ffff640 1 --2- 172.21.15.62:0/937346594 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c480039c0 0x7f7c48003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c64842640 1 --2- 172.21.15.62:0/937346594 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c480052f0 0x7f7c4800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5f7fe640 1 -- 172.21.15.62:0/937346594 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c480039c0 msgr2=0x7f7c48003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:37.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5f7fe640 1 --2- 172.21.15.62:0/937346594 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c480039c0 0x7f7c48003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:37.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5f7fe640 1 -- 172.21.15.62:0/937346594 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c480052f0 msgr2=0x7f7c4800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:37.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5f7fe640 1 --2- 172.21.15.62:0/937346594 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c480052f0 0x7f7c4800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:37.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c5f7fe640 1 -- 172.21.15.62:0/937346594 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c48003530 con 0x7f7c48004810 2024-04-18T10:06:37.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c64842640 1 --2- 172.21.15.62:0/937346594 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c480052f0 0x7f7c4800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:37.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.089+0000 7f7c64842640 1 -- 172.21.15.62:0/937346594 reap_dead start 2024-04-18T10:06:37.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c5f7fe640 1 --2- 172.21.15.62:0/937346594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 0x7f7c48004c90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009f90 tx=0x7f7c5401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb1a8cc34bbcfb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c540b2070 con 0x7f7c48004810 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c540a5b40 con 0x7f7c48004810 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c5effd640 1 -- 172.21.15.62:0/937346594 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c54009c10 con 0x7f7c48004810 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c662a5640 1 -- 172.21.15.62:0/937346594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 msgr2=0x7f7c48004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c662a5640 1 --2- 172.21.15.62:0/937346594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 0x7f7c48004c90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009f90 tx=0x7f7c5401cc40 comp rx=0 tx=0).stop 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c662a5640 1 -- 172.21.15.62:0/937346594 shutdown_connections 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c662a5640 1 --2- 172.21.15.62:0/937346594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 0x7f7c48004c90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c662a5640 1 -- 172.21.15.62:0/937346594 >> 172.21.15.62:0/937346594 conn(0x7f7c600b9390 msgr2=0x7f7c600b97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c662a5640 1 -- 172.21.15.62:0/937346594 shutdown_connections 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c662a5640 1 -- 172.21.15.62:0/937346594 wait complete. 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c662a5640 1 Processor -- start 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.093+0000 7f7c662a5640 1 -- start start 2024-04-18T10:06:37.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c662a5640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 0x7f7c6009dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:37.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c662a5640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c600b0890 0x7f7c6009c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:37.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c662a5640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c60070640 0x7f7c601e49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:37.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c662a5640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7c60149390 con 0x7f7c48004810 2024-04-18T10:06:37.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c662a5640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7c60149210 con 0x7f7c600b0890 2024-04-18T10:06:37.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c662a5640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7c60149510 con 0x7f7c60070640 2024-04-18T10:06:37.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 0x7f7c6009dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:37.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 0x7f7c6009dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41062/0 (socket says 172.21.15.62:41062) 2024-04-18T10:06:37.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5ffff640 1 -- 172.21.15.62:0/1934820574 learned_addr learned my addr 172.21.15.62:0/1934820574 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:37.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c64842640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c60070640 0x7f7c601e49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:37.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5f7fe640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c600b0890 0x7f7c6009c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:37.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5ffff640 1 -- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c60070640 msgr2=0x7f7c601e49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:37.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5ffff640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c60070640 0x7f7c601e49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:37.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5ffff640 1 -- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c600b0890 msgr2=0x7f7c6009c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:37.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5ffff640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c600b0890 0x7f7c6009c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:37.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5ffff640 1 -- 172.21.15.62:0/1934820574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c60070e00 con 0x7f7c48004810 2024-04-18T10:06:37.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5ffff640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 0x7f7c6009dee0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800e4d0 tx=0x7f7c5800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:37.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5cff9640 1 -- 172.21.15.62:0/1934820574 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c580053a0 con 0x7f7c48004810 2024-04-18T10:06:37.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5cff9640 1 -- 172.21.15.62:0/1934820574 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c58018070 con 0x7f7c48004810 2024-04-18T10:06:37.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c662a5640 1 -- 172.21.15.62:0/1934820574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c6009d6e0 con 0x7f7c48004810 2024-04-18T10:06:37.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5cff9640 1 -- 172.21.15.62:0/1934820574 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c58003820 con 0x7f7c48004810 2024-04-18T10:06:37.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c662a5640 1 -- 172.21.15.62:0/1934820574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c60070ad0 con 0x7f7c48004810 2024-04-18T10:06:37.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c662a5640 1 -- 172.21.15.62:0/1934820574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c60068b90 con 0x7f7c48004810 2024-04-18T10:06:37.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5cff9640 1 -- 172.21.15.62:0/1934820574 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7c580039c0 con 0x7f7c48004810 2024-04-18T10:06:37.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5cff9640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7c4c068380 0x7f7c4c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:37.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.097+0000 7f7c5cff9640 1 -- 172.21.15.62:0/1934820574 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5227+0+0 (secure 0 0 0) 0x7f7c58086e00 con 0x7f7c48004810 2024-04-18T10:06:37.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.101+0000 7f7c5f7fe640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7c4c068380 0x7f7c4c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:37.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.101+0000 7f7c5f7fe640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7c4c068380 0x7f7c4c06a840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7c540af000 tx=0x7f7c540a7040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:37.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.101+0000 7f7c5cff9640 1 -- 172.21.15.62:0/1934820574 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7c58057710 con 0x7f7c48004810 2024-04-18T10:06:37.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.261+0000 7f7c662a5640 1 -- 172.21.15.62:0/1934820574 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f7c601d4ff0 con 0x7f7c48004810 2024-04-18T10:06:37.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.929+0000 7f7c5cff9640 1 -- 172.21.15.62:0/1934820574 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v28) ==== 170+0+0 (secure 0 0 0) 0x7f7c58057010 con 0x7f7c48004810 2024-04-18T10:06:37.931 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' created 2024-04-18T10:06:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.933+0000 7f7c662a5640 1 -- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7c4c068380 msgr2=0x7f7c4c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.933+0000 7f7c662a5640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7c4c068380 0x7f7c4c06a840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7c540af000 tx=0x7f7c540a7040 comp rx=0 tx=0).stop 2024-04-18T10:06:37.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.933+0000 7f7c662a5640 1 -- 172.21.15.62:0/1934820574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 msgr2=0x7f7c6009dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.933+0000 7f7c662a5640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 0x7f7c6009dee0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800e4d0 tx=0x7f7c5800e990 comp rx=0 tx=0).stop 2024-04-18T10:06:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.933+0000 7f7c662a5640 1 -- 172.21.15.62:0/1934820574 shutdown_connections 2024-04-18T10:06:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.933+0000 7f7c662a5640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7c4c068380 0x7f7c4c06a840 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.933+0000 7f7c662a5640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c60070640 0x7f7c601e49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.933+0000 7f7c662a5640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c600b0890 0x7f7c6009c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.933+0000 7f7c662a5640 1 --2- 172.21.15.62:0/1934820574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7c48004810 0x7f7c6009dee0 secure :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7c5800e4d0 tx=0x7f7c5800e990 comp rx=0 tx=0).stop 2024-04-18T10:06:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.933+0000 7f7c662a5640 1 -- 172.21.15.62:0/1934820574 >> 172.21.15.62:0/1934820574 conn(0x7f7c600b9390 msgr2=0x7f7c600af370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.937+0000 7f7c662a5640 1 -- 172.21.15.62:0/1934820574 shutdown_connections 2024-04-18T10:06:37.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:37.937+0000 7f7c662a5640 1 -- 172.21.15.62:0/1934820574 wait complete. 2024-04-18T10:06:37.972 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-18T10:06:38.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f387d2d8640 1 Processor -- start 2024-04-18T10:06:38.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f387d2d8640 1 -- start start 2024-04-18T10:06:38.088 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f387d2d8640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f387806e490 0x7f3878151420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:38.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f387d2d8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f387806d4f0 con 0x7f3878146690 2024-04-18T10:06:38.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f387d2d8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f387806d7d0 con 0x7f38780b4c90 2024-04-18T10:06:38.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f387d2d8640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f387806d2a0 con 0x7f387806e490 2024-04-18T10:06:38.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f38767fc640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3878146690 0x7f387809e090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:33872/0 (socket says 172.21.15.62:33872) 2024-04-18T10:06:38.089 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f38767fc640 1 -- 172.21.15.62:0/2693101890 learned_addr learned my addr 172.21.15.62:0/2693101890 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1616298432 0 0) 0x7f387806d4f0 con 0x7f3878146690 2024-04-18T10:06:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3860003530 con 0x7f3878146690 2024-04-18T10:06:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f38777fe640 1 --2- 172.21.15.62:0/2693101890 >> v2:172.21.15.145:3300/0 conn(0x7f387806e490 0x7f3878151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3605985451 0 0) 0x7f3860003530 con 0x7f3878146690 2024-04-18T10:06:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 >> v2:172.21.15.145:3300/0 conn(0x7f387806e490 msgr2=0x7f3878151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 --2- 172.21.15.62:0/2693101890 >> v2:172.21.15.145:3300/0 conn(0x7f387806e490 0x7f3878151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 >> v1:172.21.15.145:6789/0 conn(0x7f38780b4c90 legacy=0x7f3878145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f387809c570 con 0x7f3878146690 2024-04-18T10:06:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f3868001680 con 0x7f3878146690 2024-04-18T10:06:38.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 >> v1:172.21.15.62:6789/0 conn(0x7f3878146690 legacy=0x7f387809e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 --2- 172.21.15.62:0/2693101890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38600039c0 0x7f3860003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 --2- 172.21.15.62:0/2693101890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3860004810 0x7f3860004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 --2- 172.21.15.62:0/2693101890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38600051d0 0x7f386000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f387806d2a0 con 0x7f3860004810 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f386000c170 con 0x7f38600051d0 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f386000c370 con 0x7f38600039c0 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f38767fc640 1 --2- 172.21.15.62:0/2693101890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3860004810 0x7f3860004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f38777fe640 1 --2- 172.21.15.62:0/2693101890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38600051d0 0x7f386000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f3876ffd640 1 --2- 172.21.15.62:0/2693101890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38600039c0 0x7f3860003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f38767fc640 1 -- 172.21.15.62:0/2693101890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38600039c0 msgr2=0x7f3860003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f38767fc640 1 --2- 172.21.15.62:0/2693101890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38600039c0 0x7f3860003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f38767fc640 1 -- 172.21.15.62:0/2693101890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38600051d0 msgr2=0x7f386000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f38767fc640 1 --2- 172.21.15.62:0/2693101890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38600051d0 0x7f386000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f38767fc640 1 -- 172.21.15.62:0/2693101890 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3860003530 con 0x7f3860004810 2024-04-18T10:06:38.092 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f38777fe640 1 --2- 172.21.15.62:0/2693101890 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38600051d0 0x7f386000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:38.093 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.089+0000 7f38777fe640 1 -- 172.21.15.62:0/2693101890 reap_dead start 2024-04-18T10:06:38.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f38767fc640 1 --2- 172.21.15.62:0/2693101890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3860004810 0x7f3860004c90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f386800c240 tx=0x7f386801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24bfb3283817ea8e server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:38.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38680ae070 con 0x7f3860004810 2024-04-18T10:06:38.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3868005ae0 con 0x7f3860004810 2024-04-18T10:06:38.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f3875ffb640 1 -- 172.21.15.62:0/2693101890 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3868007d20 con 0x7f3860004810 2024-04-18T10:06:38.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 -- 172.21.15.62:0/2693101890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3860004810 msgr2=0x7f3860004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:38.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 --2- 172.21.15.62:0/2693101890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3860004810 0x7f3860004c90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f386800c240 tx=0x7f386801cc40 comp rx=0 tx=0).stop 2024-04-18T10:06:38.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 -- 172.21.15.62:0/2693101890 shutdown_connections 2024-04-18T10:06:38.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 --2- 172.21.15.62:0/2693101890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3860004810 0x7f3860004c90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:38.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 -- 172.21.15.62:0/2693101890 >> 172.21.15.62:0/2693101890 conn(0x7f38780b9690 msgr2=0x7f38780b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:38.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 -- 172.21.15.62:0/2693101890 shutdown_connections 2024-04-18T10:06:38.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 -- 172.21.15.62:0/2693101890 wait complete. 2024-04-18T10:06:38.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 Processor -- start 2024-04-18T10:06:38.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 -- start start 2024-04-18T10:06:38.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3860004810 0x7f38780707f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:38.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3878071340 0x7f3878072910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:38.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3878146c90 0x7f38781e1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:38.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3878149220 con 0x7f3878071340 2024-04-18T10:06:38.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f38781490a0 con 0x7f3878146c90 2024-04-18T10:06:38.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f387d2d8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f38781493a0 con 0x7f3860004810 2024-04-18T10:06:38.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f38767fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3878071340 0x7f3878072910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:38.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f38767fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3878071340 0x7f3878072910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41082/0 (socket says 172.21.15.62:41082) 2024-04-18T10:06:38.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f38767fc640 1 -- 172.21.15.62:0/17750420 learned_addr learned my addr 172.21.15.62:0/17750420 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:38.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f3876ffd640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3860004810 0x7f38780707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:38.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f38777fe640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3878146c90 0x7f38781e1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:38.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f38767fc640 1 -- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3860004810 msgr2=0x7f38780707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f38767fc640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3860004810 0x7f38780707f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f38767fc640 1 -- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3878146c90 msgr2=0x7f38781e1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f38767fc640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3878146c90 0x7f38781e1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f38767fc640 1 -- 172.21.15.62:0/17750420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38780722a0 con 0x7f3878071340 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.093+0000 7f3876ffd640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3860004810 0x7f38780707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.097+0000 7f38767fc640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3878071340 0x7f3878072910 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f386801ca70 tx=0x7f3860005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.097+0000 7f385f7fe640 1 -- 172.21.15.62:0/17750420 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38680b8070 con 0x7f3878071340 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.097+0000 7f385f7fe640 1 -- 172.21.15.62:0/17750420 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38680049f0 con 0x7f3878071340 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.097+0000 7f385f7fe640 1 -- 172.21.15.62:0/17750420 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38680ae040 con 0x7f3878071340 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.097+0000 7f387d2d8640 1 -- 172.21.15.62:0/17750420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3878073140 con 0x7f3878071340 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.097+0000 7f387d2d8640 1 -- 172.21.15.62:0/17750420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3878071aa0 con 0x7f3878071340 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.097+0000 7f387d2d8640 1 -- 172.21.15.62:0/17750420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3878068c00 con 0x7f3878071340 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.097+0000 7f385f7fe640 1 -- 172.21.15.62:0/17750420 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f38680bb070 con 0x7f3878071340 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.097+0000 7f385f7fe640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3864068380 0x7f386406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:38.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.097+0000 7f385f7fe640 1 -- 172.21.15.62:0/17750420 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5609+0+0 (secure 0 0 0) 0x7f386811fa40 con 0x7f3878071340 2024-04-18T10:06:38.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.097+0000 7f3876ffd640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3864068380 0x7f386406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:38.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.097+0000 7f3876ffd640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3864068380 0x7f386406a840 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f386c000c10 tx=0x7f386c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:38.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.101+0000 7f385f7fe640 1 -- 172.21.15.62:0/17750420 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3868016990 con 0x7f3878071340 2024-04-18T10:06:38.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.301+0000 7f387d2d8640 1 -- 172.21.15.62:0/17750420 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f38780b44c0 con 0x7f3878071340 2024-04-18T10:06:38.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.933+0000 7f385f7fe640 1 -- 172.21.15.62:0/17750420 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) ==== 172+0+0 (secure 0 0 0) 0x7f38680efaa0 con 0x7f3878071340 2024-04-18T10:06:38.934 INFO:teuthology.orchestra.run.smithi062.stderr:new fs with metadata pool 4 and data pool 5 2024-04-18T10:06:38.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.937+0000 7f387d2d8640 1 -- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3864068380 msgr2=0x7f386406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:38.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.937+0000 7f387d2d8640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3864068380 0x7f386406a840 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f386c000c10 tx=0x7f386c009040 comp rx=0 tx=0).stop 2024-04-18T10:06:38.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.937+0000 7f387d2d8640 1 -- 172.21.15.62:0/17750420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3878071340 msgr2=0x7f3878072910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:38.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.937+0000 7f387d2d8640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3878071340 0x7f3878072910 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f386801ca70 tx=0x7f3860005840 comp rx=0 tx=0).stop 2024-04-18T10:06:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.937+0000 7f387d2d8640 1 -- 172.21.15.62:0/17750420 shutdown_connections 2024-04-18T10:06:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.937+0000 7f387d2d8640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3864068380 0x7f386406a840 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.937+0000 7f387d2d8640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3878146c90 0x7f38781e1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.937+0000 7f387d2d8640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3878071340 0x7f3878072910 secure :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f386801ca70 tx=0x7f3860005840 comp rx=0 tx=0).stop 2024-04-18T10:06:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.937+0000 7f387d2d8640 1 --2- 172.21.15.62:0/17750420 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3860004810 0x7f38780707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.937+0000 7f387d2d8640 1 -- 172.21.15.62:0/17750420 >> 172.21.15.62:0/17750420 conn(0x7f38780b9690 msgr2=0x7f3878074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.937+0000 7f387d2d8640 1 -- 172.21.15.62:0/17750420 shutdown_connections 2024-04-18T10:06:38.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:38.937+0000 7f387d2d8640 1 -- 172.21.15.62:0/17750420 wait complete. 2024-04-18T10:06:38.975 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:39.094 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.093+0000 7f20dd0f2640 1 Processor -- start 2024-04-18T10:06:39.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.093+0000 7f20dd0f2640 1 -- start start 2024-04-18T10:06:39.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f20d80bbeb0 0x7f20d80b4170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20d80b78e0 0x7f20d80b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d80bc290 con 0x7f20d80b2a40 2024-04-18T10:06:39.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f20d80b7e90 con 0x7f20d80bbeb0 2024-04-18T10:06:39.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f20d80b6740 con 0x7f20d80b78e0 2024-04-18T10:06:39.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20d6d9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f20d80bbeb0 0x7f20d80b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20d6d9d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f20d80bbeb0 0x7f20d80b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41092/0 (socket says 172.21.15.62:41092) 2024-04-18T10:06:39.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20d6d9d640 1 -- 172.21.15.62:0/3313416763 learned_addr learned my addr 172.21.15.62:0/3313416763 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:39.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20d759e640 1 --2- 172.21.15.62:0/3313416763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20d80b78e0 0x7f20d80b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20d6d9d640 1 -- 172.21.15.62:0/3313416763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20d80b78e0 msgr2=0x7f20d80b46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:39.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20d6d9d640 1 --2- 172.21.15.62:0/3313416763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20d80b78e0 0x7f20d80b46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20d6d9d640 1 -- 172.21.15.62:0/3313416763 >> v1:172.21.15.145:6789/0 conn(0x7f20d80b2a40 legacy=0x7f20d813f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:39.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20d6d9d640 1 -- 172.21.15.62:0/3313416763 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d80b63d0 con 0x7f20d80bbeb0 2024-04-18T10:06:39.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20d6d9d640 1 --2- 172.21.15.62:0/3313416763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20d80bbeb0 0x7f20d80b4170 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009870 tx=0x7f20c801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=168cc5e98df80ac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:39.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20d5d9b640 1 -- 172.21.15.62:0/3313416763 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c80b0070 con 0x7f20d80bbeb0 2024-04-18T10:06:39.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20d5d9b640 1 -- 172.21.15.62:0/3313416763 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c80ab600 con 0x7f20d80bbeb0 2024-04-18T10:06:39.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20d5d9b640 1 -- 172.21.15.62:0/3313416763 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c80ab9d0 con 0x7f20d80bbeb0 2024-04-18T10:06:39.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 -- 172.21.15.62:0/3313416763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20d80bbeb0 msgr2=0x7f20d80b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 --2- 172.21.15.62:0/3313416763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20d80bbeb0 0x7f20d80b4170 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009870 tx=0x7f20c801c4d0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 -- 172.21.15.62:0/3313416763 shutdown_connections 2024-04-18T10:06:39.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 --2- 172.21.15.62:0/3313416763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20d80b78e0 0x7f20d80b46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 --2- 172.21.15.62:0/3313416763 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20d80bbeb0 0x7f20d80b4170 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 -- 172.21.15.62:0/3313416763 >> 172.21.15.62:0/3313416763 conn(0x7f20d8067e10 msgr2=0x7f20d80bb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:39.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 -- 172.21.15.62:0/3313416763 shutdown_connections 2024-04-18T10:06:39.100 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.097+0000 7f20dd0f2640 1 -- 172.21.15.62:0/3313416763 wait complete. 2024-04-18T10:06:39.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20dd0f2640 1 Processor -- start 2024-04-18T10:06:39.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20dd0f2640 1 -- start start 2024-04-18T10:06:39.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20dd0f2640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20d80b2a40 0x7f20d806e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20dd0f2640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20d80b78e0 0x7f20d81e3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20dd0f2640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20d80bbeb0 0x7f20d81e3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20dd0f2640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f20d8152f70 con 0x7f20d80bbeb0 2024-04-18T10:06:39.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20dd0f2640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f20d8152df0 con 0x7f20d80b2a40 2024-04-18T10:06:39.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20dd0f2640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f20d81530f0 con 0x7f20d80b78e0 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20d759e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20d80bbeb0 0x7f20d81e3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20d759e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20d80bbeb0 0x7f20d81e3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41096/0 (socket says 172.21.15.62:41096) 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20d759e640 1 -- 172.21.15.62:0/1611939379 learned_addr learned my addr 172.21.15.62:0/1611939379 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20d659c640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20d80b78e0 0x7f20d81e3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20d6d9d640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20d80b2a40 0x7f20d806e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20d759e640 1 -- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20d80b78e0 msgr2=0x7f20d81e3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20d759e640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20d80b78e0 0x7f20d81e3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20d759e640 1 -- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20d80b2a40 msgr2=0x7f20d806e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20d759e640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20d80b2a40 0x7f20d806e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20d759e640 1 -- 172.21.15.62:0/1611939379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d8137380 con 0x7f20d80bbeb0 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20d6d9d640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20d80b2a40 0x7f20d806e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20d759e640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20d80bbeb0 0x7f20d81e3c90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f20d0007e40 tx=0x7f20d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20bf7fe640 1 -- 172.21.15.62:0/1611939379 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0014620 con 0x7f20d80bbeb0 2024-04-18T10:06:39.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20bf7fe640 1 -- 172.21.15.62:0/1611939379 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20d0019070 con 0x7f20d80bbeb0 2024-04-18T10:06:39.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20bf7fe640 1 -- 172.21.15.62:0/1611939379 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0014a20 con 0x7f20d80bbeb0 2024-04-18T10:06:39.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20dd0f2640 1 -- 172.21.15.62:0/1611939379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d81417d0 con 0x7f20d80bbeb0 2024-04-18T10:06:39.104 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20dd0f2640 1 -- 172.21.15.62:0/1611939379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d806ea50 con 0x7f20d80bbeb0 2024-04-18T10:06:39.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.101+0000 7f20dd0f2640 1 -- 172.21.15.62:0/1611939379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20d8139b00 con 0x7f20d80bbeb0 2024-04-18T10:06:39.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.105+0000 7f20bf7fe640 1 -- 172.21.15.62:0/1611939379 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f20d0014bc0 con 0x7f20d80bbeb0 2024-04-18T10:06:39.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.105+0000 7f20bf7fe640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f20c4068380 0x7f20c406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.105+0000 7f20bf7fe640 1 -- 172.21.15.62:0/1611939379 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5709+0+0 (secure 0 0 0) 0x7f20d0087190 con 0x7f20d80bbeb0 2024-04-18T10:06:39.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.105+0000 7f20d6d9d640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f20c4068380 0x7f20c406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.105 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.105+0000 7f20d6d9d640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f20c4068380 0x7f20c406a840 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f20c80ad000 tx=0x7f20c80a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:39.108 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.109+0000 7f20bf7fe640 1 -- 172.21.15.62:0/1611939379 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f20d00577e0 con 0x7f20d80bbeb0 2024-04-18T10:06:39.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.277+0000 7f20dd0f2640 1 -- 172.21.15.62:0/1611939379 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f20d8069230 con 0x7f20d80bbeb0 2024-04-18T10:06:39.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20bf7fe640 1 -- 172.21.15.62:0/1611939379 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+16670 (secure 0 0 0) 0x7f20d00570e0 con 0x7f20d80bbeb0 2024-04-18T10:06:39.280 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:39.280 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":29,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:38.304585+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:06:36.802689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-18T10:06:37.269237+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:39.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20dd0f2640 1 -- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f20c4068380 msgr2=0x7f20c406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20dd0f2640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f20c4068380 0x7f20c406a840 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f20c80ad000 tx=0x7f20c80a6000 comp rx=0 tx=0).stop 2024-04-18T10:06:39.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20dd0f2640 1 -- 172.21.15.62:0/1611939379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20d80bbeb0 msgr2=0x7f20d81e3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20dd0f2640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20d80bbeb0 0x7f20d81e3c90 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f20d0007e40 tx=0x7f20d000e4b0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20dd0f2640 1 -- 172.21.15.62:0/1611939379 shutdown_connections 2024-04-18T10:06:39.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20dd0f2640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f20c4068380 0x7f20c406a840 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20dd0f2640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f20d80bbeb0 0x7f20d81e3c90 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20dd0f2640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f20d80b78e0 0x7f20d81e3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20dd0f2640 1 --2- 172.21.15.62:0/1611939379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20d80b2a40 0x7f20d806e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20dd0f2640 1 -- 172.21.15.62:0/1611939379 >> 172.21.15.62:0/1611939379 conn(0x7f20d8067e10 msgr2=0x7f20d8137690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:39.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20dd0f2640 1 -- 172.21.15.62:0/1611939379 shutdown_connections 2024-04-18T10:06:39.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.281+0000 7f20dd0f2640 1 -- 172.21.15.62:0/1611939379 wait complete. 2024-04-18T10:06:39.299 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:39.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa258fc640 1 Processor -- start 2024-04-18T10:06:39.424 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa258fc640 1 -- start start 2024-04-18T10:06:39.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa258fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faa200b6260 0x7faa20148260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa258fc640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faa201487a0 0x7faa201419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa258fc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa200b6640 0x7faa200b5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa258fc640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7faa20148cb0 con 0x7faa200b6260 2024-04-18T10:06:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa258fc640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7faa200b4f90 con 0x7faa201487a0 2024-04-18T10:06:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa258fc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7faa200b7850 con 0x7faa200b6640 2024-04-18T10:06:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa1effd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faa200b6260 0x7faa20148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa1effd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faa200b6260 0x7faa20148260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41100/0 (socket says 172.21.15.62:41100) 2024-04-18T10:06:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa1effd640 1 -- 172.21.15.62:0/4034361777 learned_addr learned my addr 172.21.15.62:0/4034361777 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa1f7fe640 1 --2- 172.21.15.62:0/4034361777 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa200b6640 0x7faa200b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa1e7fc640 1 --2- 172.21.15.62:0/4034361777 >> v2:172.21.15.145:3300/0 conn(0x7faa201487a0 0x7faa201419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa1effd640 1 -- 172.21.15.62:0/4034361777 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa200b6640 msgr2=0x7faa200b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa1effd640 1 --2- 172.21.15.62:0/4034361777 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa200b6640 0x7faa200b5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa1effd640 1 -- 172.21.15.62:0/4034361777 >> v2:172.21.15.145:3300/0 conn(0x7faa201487a0 msgr2=0x7faa201419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa1effd640 1 --2- 172.21.15.62:0/4034361777 >> v2:172.21.15.145:3300/0 conn(0x7faa201487a0 0x7faa201419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.425+0000 7faa1effd640 1 -- 172.21.15.62:0/4034361777 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa200b4c20 con 0x7faa200b6260 2024-04-18T10:06:39.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1effd640 1 --2- 172.21.15.62:0/4034361777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa200b6260 0x7faa20148260 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7faa20131660 tx=0x7faa1001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35fe46e3b9c7e497 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:39.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1dffb640 1 -- 172.21.15.62:0/4034361777 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa100b1070 con 0x7faa200b6260 2024-04-18T10:06:39.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1dffb640 1 -- 172.21.15.62:0/4034361777 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa100ac600 con 0x7faa200b6260 2024-04-18T10:06:39.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1dffb640 1 -- 172.21.15.62:0/4034361777 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa100ac9d0 con 0x7faa200b6260 2024-04-18T10:06:39.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 -- 172.21.15.62:0/4034361777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa200b6260 msgr2=0x7faa20148260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 --2- 172.21.15.62:0/4034361777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa200b6260 0x7faa20148260 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7faa20131660 tx=0x7faa1001c400 comp rx=0 tx=0).stop 2024-04-18T10:06:39.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 -- 172.21.15.62:0/4034361777 shutdown_connections 2024-04-18T10:06:39.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 --2- 172.21.15.62:0/4034361777 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa200b6640 0x7faa200b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 --2- 172.21.15.62:0/4034361777 >> v2:172.21.15.145:3300/0 conn(0x7faa201487a0 0x7faa201419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 --2- 172.21.15.62:0/4034361777 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa200b6260 0x7faa20148260 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 -- 172.21.15.62:0/4034361777 >> 172.21.15.62:0/4034361777 conn(0x7faa200bd5c0 msgr2=0x7faa200bd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:39.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 -- 172.21.15.62:0/4034361777 shutdown_connections 2024-04-18T10:06:39.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 -- 172.21.15.62:0/4034361777 wait complete. 2024-04-18T10:06:39.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 Processor -- start 2024-04-18T10:06:39.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 -- start start 2024-04-18T10:06:39.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa200b6260 0x7faa20142750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa200b6640 0x7faa201433a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa201487a0 0x7faa201438e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7faa2014bea0 con 0x7faa200b6260 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7faa2014bd20 con 0x7faa201487a0 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7faa2014c020 con 0x7faa200b6640 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa200b6260 0x7faa20142750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1effd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa200b6260 0x7faa20142750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41110/0 (socket says 172.21.15.62:41110) 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1effd640 1 -- 172.21.15.62:0/2350626648 learned_addr learned my addr 172.21.15.62:0/2350626648 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1effd640 1 -- 172.21.15.62:0/2350626648 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa200b6640 msgr2=0x7faa201433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1effd640 1 --2- 172.21.15.62:0/2350626648 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa200b6640 0x7faa201433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1effd640 1 -- 172.21.15.62:0/2350626648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa201487a0 msgr2=0x7faa201438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1effd640 1 --2- 172.21.15.62:0/2350626648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa201487a0 0x7faa201438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1effd640 1 -- 172.21.15.62:0/2350626648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa201458a0 con 0x7faa200b6260 2024-04-18T10:06:39.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa1effd640 1 --2- 172.21.15.62:0/2350626648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa200b6260 0x7faa20142750 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7faa1001aab0 tx=0x7faa1001ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:39.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa07fff640 1 -- 172.21.15.62:0/2350626648 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa100b1040 con 0x7faa200b6260 2024-04-18T10:06:39.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa07fff640 1 -- 172.21.15.62:0/2350626648 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa100bc070 con 0x7faa200b6260 2024-04-18T10:06:39.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa07fff640 1 -- 172.21.15.62:0/2350626648 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa100b7670 con 0x7faa200b6260 2024-04-18T10:06:39.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 -- 172.21.15.62:0/2350626648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa20144a10 con 0x7faa200b6260 2024-04-18T10:06:39.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.429+0000 7faa258fc640 1 -- 172.21.15.62:0/2350626648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa2013f4f0 con 0x7faa200b6260 2024-04-18T10:06:39.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.433+0000 7faa258fc640 1 -- 172.21.15.62:0/2350626648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa20073bf0 con 0x7faa200b6260 2024-04-18T10:06:39.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.433+0000 7faa07fff640 1 -- 172.21.15.62:0/2350626648 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faa100ac4c0 con 0x7faa200b6260 2024-04-18T10:06:39.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.433+0000 7faa07fff640 1 --2- 172.21.15.62:0/2350626648 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faa0c0683b0 0x7faa0c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.433+0000 7faa07fff640 1 -- 172.21.15.62:0/2350626648 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5709+0+0 (secure 0 0 0) 0x7faa101209f0 con 0x7faa200b6260 2024-04-18T10:06:39.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.433+0000 7faa1e7fc640 1 --2- 172.21.15.62:0/2350626648 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faa0c0683b0 0x7faa0c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.433+0000 7faa1e7fc640 1 --2- 172.21.15.62:0/2350626648 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faa0c0683b0 0x7faa0c06a870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faa14006aa0 tx=0x7faa14006a30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:39.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.437+0000 7faa07fff640 1 -- 172.21.15.62:0/2350626648 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7faa10016110 con 0x7faa200b6260 2024-04-18T10:06:39.598 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.597+0000 7faa258fc640 1 -- 172.21.15.62:0/2350626648 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa201456a0 con 0x7faa200b6260 2024-04-18T10:06:39.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa07fff640 1 -- 172.21.15.62:0/2350626648 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+16670 (secure 0 0 0) 0x7faa100f1040 con 0x7faa200b6260 2024-04-18T10:06:39.601 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:39.601 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":29,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:38.304585+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:06:36.802689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-18T10:06:37.269237+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:39.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa258fc640 1 -- 172.21.15.62:0/2350626648 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faa0c0683b0 msgr2=0x7faa0c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa258fc640 1 --2- 172.21.15.62:0/2350626648 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faa0c0683b0 0x7faa0c06a870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faa14006aa0 tx=0x7faa14006a30 comp rx=0 tx=0).stop 2024-04-18T10:06:39.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa258fc640 1 -- 172.21.15.62:0/2350626648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa200b6260 msgr2=0x7faa20142750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa258fc640 1 --2- 172.21.15.62:0/2350626648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa200b6260 0x7faa20142750 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7faa1001aab0 tx=0x7faa1001ac40 comp rx=0 tx=0).stop 2024-04-18T10:06:39.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa258fc640 1 -- 172.21.15.62:0/2350626648 shutdown_connections 2024-04-18T10:06:39.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa258fc640 1 --2- 172.21.15.62:0/2350626648 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faa0c0683b0 0x7faa0c06a870 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa258fc640 1 --2- 172.21.15.62:0/2350626648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa201487a0 0x7faa201438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa258fc640 1 --2- 172.21.15.62:0/2350626648 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa200b6640 0x7faa201433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa258fc640 1 --2- 172.21.15.62:0/2350626648 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faa200b6260 0x7faa20142750 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa258fc640 1 -- 172.21.15.62:0/2350626648 >> 172.21.15.62:0/2350626648 conn(0x7faa200bd5c0 msgr2=0x7faa200b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:39.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa258fc640 1 -- 172.21.15.62:0/2350626648 shutdown_connections 2024-04-18T10:06:39.604 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.601+0000 7faa258fc640 1 -- 172.21.15.62:0/2350626648 wait complete. 2024-04-18T10:06:39.619 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-18T10:06:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.741+0000 7ff084667640 1 Processor -- start 2024-04-18T10:06:39.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.741+0000 7ff084667640 1 -- start start 2024-04-18T10:06:39.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.741+0000 7ff084667640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff07c14b190 0x7ff07c147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.741+0000 7ff084667640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff07c14b620 con 0x7ff07c0b3eb0 2024-04-18T10:06:39.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.741+0000 7ff084667640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff07c14b900 con 0x7ff07c09b9d0 2024-04-18T10:06:39.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.741+0000 7ff084667640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7ff07c0b4290 con 0x7ff07c14b190 2024-04-18T10:06:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.741+0000 7ff081c02640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff07c14b190 0x7ff07c147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.741+0000 7ff081c02640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff07c14b190 0x7ff07c147ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41120/0 (socket says 172.21.15.62:41120) 2024-04-18T10:06:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.741+0000 7ff081c02640 1 -- 172.21.15.62:0/622802099 learned_addr learned my addr 172.21.15.62:0/622802099 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.741+0000 7ff081c02640 1 -- 172.21.15.62:0/622802099 >> v1:172.21.15.145:6789/0 conn(0x7ff07c09b9d0 legacy=0x7ff07c152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.741+0000 7ff081c02640 1 -- 172.21.15.62:0/622802099 >> v1:172.21.15.62:6789/0 conn(0x7ff07c0b3eb0 legacy=0x7ff07c0bdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.741+0000 7ff081c02640 1 -- 172.21.15.62:0/622802099 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff07c0b4460 con 0x7ff07c14b190 2024-04-18T10:06:39.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff081c02640 1 --2- 172.21.15.62:0/622802099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff07c14b190 0x7ff07c147ba0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff070009e50 tx=0x7ff07001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df145697e44638b server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:39.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff081401640 1 -- 172.21.15.62:0/622802099 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0700a5620 con 0x7ff07c14b190 2024-04-18T10:06:39.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff081401640 1 -- 172.21.15.62:0/622802099 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0700a57c0 con 0x7ff07c14b190 2024-04-18T10:06:39.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff081401640 1 -- 172.21.15.62:0/622802099 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0700a5b90 con 0x7ff07c14b190 2024-04-18T10:06:39.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 -- 172.21.15.62:0/622802099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff07c14b190 msgr2=0x7ff07c147ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 --2- 172.21.15.62:0/622802099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff07c14b190 0x7ff07c147ba0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff070009e50 tx=0x7ff07001c960 comp rx=0 tx=0).stop 2024-04-18T10:06:39.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 -- 172.21.15.62:0/622802099 shutdown_connections 2024-04-18T10:06:39.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 --2- 172.21.15.62:0/622802099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff07c14b190 0x7ff07c147ba0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 -- 172.21.15.62:0/622802099 >> 172.21.15.62:0/622802099 conn(0x7ff07c067e10 msgr2=0x7ff07c0bd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:39.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 -- 172.21.15.62:0/622802099 shutdown_connections 2024-04-18T10:06:39.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 -- 172.21.15.62:0/622802099 wait complete. 2024-04-18T10:06:39.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 Processor -- start 2024-04-18T10:06:39.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 -- start start 2024-04-18T10:06:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff07c0b3eb0 0x7ff07c0b8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff07c13b460 0x7ff07c1e4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff07c0ba4f0 0x7ff07c137a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c156c90 con 0x7ff07c13b460 2024-04-18T10:06:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7ff07c14c0e0 con 0x7ff07c0b3eb0 2024-04-18T10:06:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff084667640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7ff07c156e10 con 0x7ff07c0ba4f0 2024-04-18T10:06:39.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff081c02640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff07c13b460 0x7ff07c1e4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff081c02640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff07c13b460 0x7ff07c1e4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:41122/0 (socket says 172.21.15.62:41122) 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff081c02640 1 -- 172.21.15.62:0/2458881768 learned_addr learned my addr 172.21.15.62:0/2458881768 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff082403640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff07c0b3eb0 0x7ff07c0b8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff082c04640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff07c0ba4f0 0x7ff07c137a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.745+0000 7ff081c02640 1 -- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff07c0ba4f0 msgr2=0x7ff07c137a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff081c02640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff07c0ba4f0 0x7ff07c137a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff081c02640 1 -- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff07c0b3eb0 msgr2=0x7ff07c0b8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff081c02640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff07c0b3eb0 0x7ff07c0b8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff081c02640 1 -- 172.21.15.62:0/2458881768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff07c0b9a50 con 0x7ff07c13b460 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff082c04640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff07c0ba4f0 0x7ff07c137a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff081c02640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff07c13b460 0x7ff07c1e4260 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff0700af040 tx=0x7ff070019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff07affd640 1 -- 172.21.15.62:0/2458881768 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0700bc020 con 0x7ff07c13b460 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff07affd640 1 -- 172.21.15.62:0/2458881768 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0700b8690 con 0x7ff07c13b460 2024-04-18T10:06:39.749 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff084667640 1 -- 172.21.15.62:0/2458881768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff07c0b9cf0 con 0x7ff07c13b460 2024-04-18T10:06:39.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff07affd640 1 -- 172.21.15.62:0/2458881768 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0700b8a80 con 0x7ff07c13b460 2024-04-18T10:06:39.750 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff084667640 1 -- 172.21.15.62:0/2458881768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff07c1e47a0 con 0x7ff07c13b460 2024-04-18T10:06:39.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff084667640 1 -- 172.21.15.62:0/2458881768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff07c068c00 con 0x7ff07c13b460 2024-04-18T10:06:39.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff07affd640 1 -- 172.21.15.62:0/2458881768 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff0700bf020 con 0x7ff07c13b460 2024-04-18T10:06:39.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff07affd640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff068068380 0x7ff06806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:39.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff07affd640 1 -- 172.21.15.62:0/2458881768 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5709+0+0 (secure 0 0 0) 0x7ff0701219a0 con 0x7ff07c13b460 2024-04-18T10:06:39.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff082403640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff068068380 0x7ff06806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:39.751 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.749+0000 7ff082403640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff068068380 0x7ff06806a840 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff07c1e1e20 tx=0x7ff074009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:39.755 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.753+0000 7ff07affd640 1 -- 172.21.15.62:0/2458881768 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff0700c6030 con 0x7ff07c13b460 2024-04-18T10:06:39.956 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:39.957+0000 7ff084667640 1 -- 172.21.15.62:0/2458881768 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff07c147db0 con 0x7ff07c13b460 2024-04-18T10:06:40.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.965+0000 7ff07affd640 1 -- 172.21.15.62:0/2458881768 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7ff0700165e0 con 0x7ff07c13b460 2024-04-18T10:06:40.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.969+0000 7ff084667640 1 -- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff068068380 msgr2=0x7ff06806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:40.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.969+0000 7ff084667640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff068068380 0x7ff06806a840 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff07c1e1e20 tx=0x7ff074009040 comp rx=0 tx=0).stop 2024-04-18T10:06:40.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.969+0000 7ff084667640 1 -- 172.21.15.62:0/2458881768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff07c13b460 msgr2=0x7ff07c1e4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:40.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.969+0000 7ff084667640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff07c13b460 0x7ff07c1e4260 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff0700af040 tx=0x7ff070019700 comp rx=0 tx=0).stop 2024-04-18T10:06:40.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.969+0000 7ff084667640 1 -- 172.21.15.62:0/2458881768 shutdown_connections 2024-04-18T10:06:40.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.969+0000 7ff084667640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff068068380 0x7ff06806a840 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:40.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.969+0000 7ff084667640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff07c0ba4f0 0x7ff07c137a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:40.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.969+0000 7ff084667640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff07c13b460 0x7ff07c1e4260 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:40.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.969+0000 7ff084667640 1 --2- 172.21.15.62:0/2458881768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff07c0b3eb0 0x7ff07c0b8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:40.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.969+0000 7ff084667640 1 -- 172.21.15.62:0/2458881768 >> 172.21.15.62:0/2458881768 conn(0x7ff07c067e10 msgr2=0x7ff07c0b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:40.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.969+0000 7ff084667640 1 -- 172.21.15.62:0/2458881768 shutdown_connections 2024-04-18T10:06:40.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:40.969+0000 7ff084667640 1 -- 172.21.15.62:0/2458881768 wait complete. 2024-04-18T10:06:40.993 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-18T10:06:40.994 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-18T10:06:41.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa989433640 1 Processor -- start 2024-04-18T10:06:41.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa989433640 1 -- start start 2024-04-18T10:06:41.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa989433640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa9840bb1d0 0x7fa9841462b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:41.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa989433640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa98414e420 0x7fa984150810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:41.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa989433640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa984073800 con 0x7fa984073420 2024-04-18T10:06:41.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa989433640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fa9840740d0 con 0x7fa9840bb1d0 2024-04-18T10:06:41.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa989433640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fa9840ba6f0 con 0x7fa98414e420 2024-04-18T10:06:41.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa9827fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa9840bb1d0 0x7fa9841462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:41.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa9827fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa9840bb1d0 0x7fa9841462b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56348/0 (socket says 172.21.15.62:56348) 2024-04-18T10:06:41.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa9827fc640 1 -- 172.21.15.62:0/1727165440 learned_addr learned my addr 172.21.15.62:0/1727165440 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:41.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa9837fe640 1 --2- 172.21.15.62:0/1727165440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa98414e420 0x7fa984150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:41.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa9827fc640 1 -- 172.21.15.62:0/1727165440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa98414e420 msgr2=0x7fa984150810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:41.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa9827fc640 1 --2- 172.21.15.62:0/1727165440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa98414e420 0x7fa984150810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:41.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa9827fc640 1 -- 172.21.15.62:0/1727165440 >> v1:172.21.15.145:6789/0 conn(0x7fa984073420 legacy=0x7fa984145ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:41.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa9827fc640 1 -- 172.21.15.62:0/1727165440 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa984073d60 con 0x7fa9840bb1d0 2024-04-18T10:06:41.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.125+0000 7fa9827fc640 1 --2- 172.21.15.62:0/1727165440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9840bb1d0 0x7fa9841462b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa978009870 tx=0x7fa97801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf9c4181f8b3db06 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:41.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa981ffb640 1 -- 172.21.15.62:0/1727165440 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9780b0070 con 0x7fa9840bb1d0 2024-04-18T10:06:41.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa981ffb640 1 -- 172.21.15.62:0/1727165440 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9780ab600 con 0x7fa9840bb1d0 2024-04-18T10:06:41.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa981ffb640 1 -- 172.21.15.62:0/1727165440 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9780ab9d0 con 0x7fa9840bb1d0 2024-04-18T10:06:41.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 -- 172.21.15.62:0/1727165440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9840bb1d0 msgr2=0x7fa9841462b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:41.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 --2- 172.21.15.62:0/1727165440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9840bb1d0 0x7fa9841462b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa978009870 tx=0x7fa97801c400 comp rx=0 tx=0).stop 2024-04-18T10:06:41.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 -- 172.21.15.62:0/1727165440 shutdown_connections 2024-04-18T10:06:41.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 --2- 172.21.15.62:0/1727165440 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa98414e420 0x7fa984150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:41.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 --2- 172.21.15.62:0/1727165440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa9840bb1d0 0x7fa9841462b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:41.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 -- 172.21.15.62:0/1727165440 >> 172.21.15.62:0/1727165440 conn(0x7fa98409d470 msgr2=0x7fa98409d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:41.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 -- 172.21.15.62:0/1727165440 shutdown_connections 2024-04-18T10:06:41.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 -- 172.21.15.62:0/1727165440 wait complete. 2024-04-18T10:06:41.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 Processor -- start 2024-04-18T10:06:41.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 -- start start 2024-04-18T10:06:41.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa984073420 0x7fa9840b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:41.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9840bb1d0 0x7fa9841503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:41.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa98414e420 0x7fa9841e4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:41.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa9841467f0 con 0x7fa98414e420 2024-04-18T10:06:41.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fa984070ec0 con 0x7fa984073420 2024-04-18T10:06:41.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fa984146970 con 0x7fa9840bb1d0 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa9837fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa98414e420 0x7fa9841e4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa9837fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa98414e420 0x7fa9841e4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56352/0 (socket says 172.21.15.62:56352) 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa9837fe640 1 -- 172.21.15.62:0/3921241350 learned_addr learned my addr 172.21.15.62:0/3921241350 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa982ffd640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa984073420 0x7fa9840b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa9827fc640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9840bb1d0 0x7fa9841503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa9837fe640 1 -- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9840bb1d0 msgr2=0x7fa9841503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa9837fe640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9840bb1d0 0x7fa9841503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa9837fe640 1 -- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa984073420 msgr2=0x7fa9840b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa9837fe640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa984073420 0x7fa9840b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa9837fe640 1 -- 172.21.15.62:0/3921241350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9841e4540 con 0x7fa98414e420 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa982ffd640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa984073420 0x7fa9840b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa9827fc640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9840bb1d0 0x7fa9841503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa9837fe640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa98414e420 0x7fa9841e4000 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa97c007e60 tx=0x7fa97c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:41.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa96b7fe640 1 -- 172.21.15.62:0/3921241350 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c014620 con 0x7fa98414e420 2024-04-18T10:06:41.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa96b7fe640 1 -- 172.21.15.62:0/3921241350 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa97c019070 con 0x7fa98414e420 2024-04-18T10:06:41.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 -- 172.21.15.62:0/3921241350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa98406d3a0 con 0x7fa98414e420 2024-04-18T10:06:41.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa96b7fe640 1 -- 172.21.15.62:0/3921241350 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c014a00 con 0x7fa98414e420 2024-04-18T10:06:41.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.129+0000 7fa989433640 1 -- 172.21.15.62:0/3921241350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa98406d920 con 0x7fa98414e420 2024-04-18T10:06:41.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.133+0000 7fa989433640 1 -- 172.21.15.62:0/3921241350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa984069230 con 0x7fa98414e420 2024-04-18T10:06:41.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.133+0000 7fa96b7fe640 1 -- 172.21.15.62:0/3921241350 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa97c014ba0 con 0x7fa98414e420 2024-04-18T10:06:41.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.133+0000 7fa96b7fe640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa970068380 0x7fa97006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:41.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.133+0000 7fa96b7fe640 1 -- 172.21.15.62:0/3921241350 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fa97c087230 con 0x7fa98414e420 2024-04-18T10:06:41.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.133+0000 7fa982ffd640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa970068380 0x7fa97006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:41.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.133+0000 7fa982ffd640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa970068380 0x7fa97006a840 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa97400a850 tx=0x7fa974008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:41.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.137+0000 7fa96b7fe640 1 -- 172.21.15.62:0/3921241350 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa97c011260 con 0x7fa98414e420 2024-04-18T10:06:41.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.337+0000 7fa989433640 1 -- 172.21.15.62:0/3921241350 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fa984143c30 con 0x7fa98414e420 2024-04-18T10:06:41.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.969+0000 7fa96b7fe640 1 -- 172.21.15.62:0/3921241350 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7fa97c057880 con 0x7fa98414e420 2024-04-18T10:06:41.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.973+0000 7fa989433640 1 -- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa970068380 msgr2=0x7fa97006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:41.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.973+0000 7fa989433640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa970068380 0x7fa97006a840 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa97400a850 tx=0x7fa974008040 comp rx=0 tx=0).stop 2024-04-18T10:06:41.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.973+0000 7fa989433640 1 -- 172.21.15.62:0/3921241350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa98414e420 msgr2=0x7fa9841e4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:41.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.973+0000 7fa989433640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa98414e420 0x7fa9841e4000 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fa97c007e60 tx=0x7fa97c00e4b0 comp rx=0 tx=0).stop 2024-04-18T10:06:41.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.973+0000 7fa989433640 1 -- 172.21.15.62:0/3921241350 shutdown_connections 2024-04-18T10:06:41.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.973+0000 7fa989433640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa970068380 0x7fa97006a840 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:41.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.973+0000 7fa989433640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa98414e420 0x7fa9841e4000 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:41.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.973+0000 7fa989433640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9840bb1d0 0x7fa9841503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:41.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.973+0000 7fa989433640 1 --2- 172.21.15.62:0/3921241350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa984073420 0x7fa9840b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:41.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.973+0000 7fa989433640 1 -- 172.21.15.62:0/3921241350 >> 172.21.15.62:0/3921241350 conn(0x7fa98409d470 msgr2=0x7fa98406f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:41.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.973+0000 7fa989433640 1 -- 172.21.15.62:0/3921241350 shutdown_connections 2024-04-18T10:06:41.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:41.973+0000 7fa989433640 1 -- 172.21.15.62:0/3921241350 wait complete. 2024-04-18T10:06:41.996 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-18T10:06:41.996 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:42.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff68dda8640 1 Processor -- start 2024-04-18T10:06:42.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff68dda8640 1 -- start start 2024-04-18T10:06:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff68dda8640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff6880b44e0 0x7ff6880b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff68dda8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff688143bb0 0x7ff688144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff68dda8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6880ba170 con 0x7ff6880b9d90 2024-04-18T10:06:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff68dda8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7ff688145b80 con 0x7ff688143bb0 2024-04-18T10:06:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff68dda8640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7ff6881458a0 con 0x7ff6880b44e0 2024-04-18T10:06:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff687fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff688143bb0 0x7ff688144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff6877fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff6880b9d90 0x7ff6881450e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42836/0 (socket says 172.21.15.62:42836) 2024-04-18T10:06:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff687fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff688143bb0 0x7ff688144020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56364/0 (socket says 172.21.15.62:56364) 2024-04-18T10:06:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff6877fe640 1 -- 172.21.15.62:0/1997035048 learned_addr learned my addr 172.21.15.62:0/1997035048 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff67effd640 1 --2- 172.21.15.62:0/1997035048 >> v2:172.21.15.145:3300/0 conn(0x7ff6880b44e0 0x7ff6880b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff687fff640 1 -- 172.21.15.62:0/1997035048 >> v2:172.21.15.145:3300/0 conn(0x7ff6880b44e0 msgr2=0x7ff6880b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff687fff640 1 --2- 172.21.15.62:0/1997035048 >> v2:172.21.15.145:3300/0 conn(0x7ff6880b44e0 0x7ff6880b4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff687fff640 1 -- 172.21.15.62:0/1997035048 >> v1:172.21.15.62:6789/0 conn(0x7ff6880b9d90 legacy=0x7ff6881450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.125+0000 7ff687fff640 1 -- 172.21.15.62:0/1997035048 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6880b3fc0 con 0x7ff688143bb0 2024-04-18T10:06:42.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff687fff640 1 --2- 172.21.15.62:0/1997035048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff688143bb0 0x7ff688144020 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff6880ae9b0 tx=0x7ff68001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2ff051e3d6d1d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:42.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff686ffd640 1 -- 172.21.15.62:0/1997035048 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800b2070 con 0x7ff688143bb0 2024-04-18T10:06:42.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff686ffd640 1 -- 172.21.15.62:0/1997035048 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6800a5760 con 0x7ff688143bb0 2024-04-18T10:06:42.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff686ffd640 1 -- 172.21.15.62:0/1997035048 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800a5b30 con 0x7ff688143bb0 2024-04-18T10:06:42.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 -- 172.21.15.62:0/1997035048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff688143bb0 msgr2=0x7ff688144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 --2- 172.21.15.62:0/1997035048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff688143bb0 0x7ff688144020 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff6880ae9b0 tx=0x7ff68001c8a0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 -- 172.21.15.62:0/1997035048 shutdown_connections 2024-04-18T10:06:42.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 --2- 172.21.15.62:0/1997035048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff688143bb0 0x7ff688144020 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 --2- 172.21.15.62:0/1997035048 >> v2:172.21.15.145:3300/0 conn(0x7ff6880b44e0 0x7ff6880b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 -- 172.21.15.62:0/1997035048 >> 172.21.15.62:0/1997035048 conn(0x7ff68806c0d0 msgr2=0x7ff6880b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:42.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 -- 172.21.15.62:0/1997035048 shutdown_connections 2024-04-18T10:06:42.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 -- 172.21.15.62:0/1997035048 wait complete. 2024-04-18T10:06:42.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 Processor -- start 2024-04-18T10:06:42.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 -- start start 2024-04-18T10:06:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6880b44e0 0x7ff68813bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6880b9d90 0x7ff68813e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff688143bb0 0x7ff6881de950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff688152010 con 0x7ff688143bb0 2024-04-18T10:06:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7ff688151e90 con 0x7ff6880b44e0 2024-04-18T10:06:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff68dda8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7ff688152190 con 0x7ff6880b9d90 2024-04-18T10:06:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff687fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff688143bb0 0x7ff6881de950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff687fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff688143bb0 0x7ff6881de950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56372/0 (socket says 172.21.15.62:56372) 2024-04-18T10:06:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff687fff640 1 -- 172.21.15.62:0/2792457032 learned_addr learned my addr 172.21.15.62:0/2792457032 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:42.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff67effd640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6880b9d90 0x7ff68813e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff6877fe640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6880b44e0 0x7ff68813bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff687fff640 1 -- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6880b9d90 msgr2=0x7ff68813e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff687fff640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6880b9d90 0x7ff68813e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff687fff640 1 -- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6880b44e0 msgr2=0x7ff68813bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff687fff640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6880b44e0 0x7ff68813bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff687fff640 1 -- 172.21.15.62:0/2792457032 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68813b1d0 con 0x7ff688143bb0 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff67effd640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6880b9d90 0x7ff68813e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.129+0000 7ff6877fe640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6880b44e0 0x7ff68813bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.133+0000 7ff687fff640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff688143bb0 0x7ff6881de950 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff6800194b0 tx=0x7ff680019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.133+0000 7ff684ff9640 1 -- 172.21.15.62:0/2792457032 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800b2050 con 0x7ff688143bb0 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.133+0000 7ff684ff9640 1 -- 172.21.15.62:0/2792457032 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6800be070 con 0x7ff688143bb0 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.133+0000 7ff68dda8640 1 -- 172.21.15.62:0/2792457032 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff688071400 con 0x7ff688143bb0 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.133+0000 7ff684ff9640 1 -- 172.21.15.62:0/2792457032 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800ad890 con 0x7ff688143bb0 2024-04-18T10:06:42.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.133+0000 7ff68dda8640 1 -- 172.21.15.62:0/2792457032 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff68813a6f0 con 0x7ff688143bb0 2024-04-18T10:06:42.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.133+0000 7ff68dda8640 1 -- 172.21.15.62:0/2792457032 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff688068b90 con 0x7ff688143bb0 2024-04-18T10:06:42.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.133+0000 7ff684ff9640 1 -- 172.21.15.62:0/2792457032 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff6800ada30 con 0x7ff688143bb0 2024-04-18T10:06:42.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.133+0000 7ff684ff9640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff66c068380 0x7ff66c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.133+0000 7ff684ff9640 1 -- 172.21.15.62:0/2792457032 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7ff6801219b0 con 0x7ff688143bb0 2024-04-18T10:06:42.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.133+0000 7ff6877fe640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff66c068380 0x7ff66c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.133+0000 7ff6877fe640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff66c068380 0x7ff66c06a840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff68813f610 tx=0x7ff678009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:42.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.137+0000 7ff684ff9640 1 -- 172.21.15.62:0/2792457032 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff680016520 con 0x7ff688143bb0 2024-04-18T10:06:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.333+0000 7ff68dda8640 1 -- 172.21.15.62:0/2792457032 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6880b5310 con 0x7ff688143bb0 2024-04-18T10:06:42.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.333+0000 7ff684ff9640 1 -- 172.21.15.62:0/2792457032 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+3534 (secure 0 0 0) 0x7ff6800f2120 con 0x7ff688143bb0 2024-04-18T10:06:42.334 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:42.335 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4705,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1649467687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1649467687},{"type":"v1","addr":"172.21.15.145:6835","nonce":1649467687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:06:38.304481+0000","modified":"2024-04-18T10:06:41.339980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4932":{"gid":4932,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2971073685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2971073685},{"type":"v1","addr":"172.21.15.62:6833","nonce":2971073685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4932,"qdb_cluster":[4932]},"id":2}]} 2024-04-18T10:06:42.335 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-04-18T10:06:42.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.337+0000 7ff68dda8640 1 -- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff66c068380 msgr2=0x7ff66c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.337+0000 7ff68dda8640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff66c068380 0x7ff66c06a840 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff68813f610 tx=0x7ff678009040 comp rx=0 tx=0).stop 2024-04-18T10:06:42.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.337+0000 7ff68dda8640 1 -- 172.21.15.62:0/2792457032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff688143bb0 msgr2=0x7ff6881de950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.337+0000 7ff68dda8640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff688143bb0 0x7ff6881de950 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff6800194b0 tx=0x7ff680019640 comp rx=0 tx=0).stop 2024-04-18T10:06:42.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.337+0000 7ff68dda8640 1 -- 172.21.15.62:0/2792457032 shutdown_connections 2024-04-18T10:06:42.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.337+0000 7ff68dda8640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff66c068380 0x7ff66c06a840 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.337+0000 7ff68dda8640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff688143bb0 0x7ff6881de950 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.337+0000 7ff68dda8640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6880b9d90 0x7ff68813e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.337+0000 7ff68dda8640 1 --2- 172.21.15.62:0/2792457032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6880b44e0 0x7ff68813bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.337+0000 7ff68dda8640 1 -- 172.21.15.62:0/2792457032 >> 172.21.15.62:0/2792457032 conn(0x7ff68806c0d0 msgr2=0x7ff688074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:42.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.337+0000 7ff68dda8640 1 -- 172.21.15.62:0/2792457032 shutdown_connections 2024-04-18T10:06:42.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.337+0000 7ff68dda8640 1 -- 172.21.15.62:0/2792457032 wait complete. 2024-04-18T10:06:42.352 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:42.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.473+0000 7fb379b56640 1 Processor -- start 2024-04-18T10:06:42.474 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.473+0000 7fb379b56640 1 -- start start 2024-04-18T10:06:42.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.473+0000 7fb379b56640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb3740b6640 0x7fb3741419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.473+0000 7fb379b56640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3740b7850 con 0x7fb374141ee0 2024-04-18T10:06:42.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.473+0000 7fb379b56640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3740b4c20 con 0x7fb3740b6260 2024-04-18T10:06:42.475 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.473+0000 7fb379b56640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fb374148cb0 con 0x7fb3740b6640 2024-04-18T10:06:42.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.473+0000 7fb373fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb374141ee0 0x7fb3740b5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42850/0 (socket says 172.21.15.62:42850) 2024-04-18T10:06:42.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.473+0000 7fb373fff640 1 -- 172.21.15.62:0/3376029430 learned_addr learned my addr 172.21.15.62:0/3376029430 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:42.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb372ffd640 1 --2- 172.21.15.62:0/3376029430 >> v2:172.21.15.145:3300/0 conn(0x7fb3740b6640 0x7fb3741419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1494012482 0 0) 0x7fb3740b7850 con 0x7fb374141ee0 2024-04-18T10:06:42.476 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb35c003530 con 0x7fb374141ee0 2024-04-18T10:06:42.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 591247992 0 0) 0x7fb35c003530 con 0x7fb374141ee0 2024-04-18T10:06:42.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 >> v2:172.21.15.145:3300/0 conn(0x7fb3740b6640 msgr2=0x7fb3741419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:42.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 --2- 172.21.15.62:0/3376029430 >> v2:172.21.15.145:3300/0 conn(0x7fb3740b6640 0x7fb3741419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 >> v1:172.21.15.145:6789/0 conn(0x7fb3740b6260 legacy=0x7fb374148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb374148970 con 0x7fb374141ee0 2024-04-18T10:06:42.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fb36c001a00 con 0x7fb374141ee0 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 >> v1:172.21.15.62:6789/0 conn(0x7fb374141ee0 legacy=0x7fb3740b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 --2- 172.21.15.62:0/3376029430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb35c003a10 0x7fb35c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 --2- 172.21.15.62:0/3376029430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb35c004810 0x7fb35c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 --2- 172.21.15.62:0/3376029430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb35c0052f0 0x7fb35c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb374148cb0 con 0x7fb35c003a10 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fb35c00c1f0 con 0x7fb35c0052f0 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fb35c00c3f0 con 0x7fb35c004810 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3737fe640 1 --2- 172.21.15.62:0/3376029430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb35c003a10 0x7fb35c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb373fff640 1 --2- 172.21.15.62:0/3376029430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb35c0052f0 0x7fb35c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb372ffd640 1 --2- 172.21.15.62:0/3376029430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb35c004810 0x7fb35c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3737fe640 1 -- 172.21.15.62:0/3376029430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb35c004810 msgr2=0x7fb35c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3737fe640 1 --2- 172.21.15.62:0/3376029430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb35c004810 0x7fb35c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3737fe640 1 -- 172.21.15.62:0/3376029430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb35c0052f0 msgr2=0x7fb35c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:42.478 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3737fe640 1 --2- 172.21.15.62:0/3376029430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb35c0052f0 0x7fb35c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3737fe640 1 -- 172.21.15.62:0/3376029430 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb35c003530 con 0x7fb35c003a10 2024-04-18T10:06:42.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb373fff640 1 --2- 172.21.15.62:0/3376029430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb35c0052f0 0x7fb35c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:42.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb373fff640 1 -- 172.21.15.62:0/3376029430 reap_dead start 2024-04-18T10:06:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3737fe640 1 --2- 172.21.15.62:0/3376029430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb35c003a10 0x7fb35c003e30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb364001780 tx=0x7fb36401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817229fd567eea74 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3640013a0 con 0x7fb35c003a10 2024-04-18T10:06:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3640052f0 con 0x7fb35c003a10 2024-04-18T10:06:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.477+0000 7fb3727fc640 1 -- 172.21.15.62:0/3376029430 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364003840 con 0x7fb35c003a10 2024-04-18T10:06:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 -- 172.21.15.62:0/3376029430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb35c003a10 msgr2=0x7fb35c003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 --2- 172.21.15.62:0/3376029430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb35c003a10 0x7fb35c003e30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb364001780 tx=0x7fb36401c400 comp rx=0 tx=0).stop 2024-04-18T10:06:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 -- 172.21.15.62:0/3376029430 shutdown_connections 2024-04-18T10:06:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 --2- 172.21.15.62:0/3376029430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb35c003a10 0x7fb35c003e30 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 -- 172.21.15.62:0/3376029430 >> 172.21.15.62:0/3376029430 conn(0x7fb3740bd5c0 msgr2=0x7fb3740bd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 -- 172.21.15.62:0/3376029430 shutdown_connections 2024-04-18T10:06:42.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 -- 172.21.15.62:0/3376029430 wait complete. 2024-04-18T10:06:42.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 Processor -- start 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 -- start start 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb35c003a10 0x7fb3741d6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb374149170 0x7fb3741e43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3741e4910 0x7fb37413fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb37414b300 con 0x7fb3741e4910 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fb3740b5e20 con 0x7fb374149170 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fb37414b480 con 0x7fb35c003a10 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb373fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3741e4910 0x7fb37413fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb373fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3741e4910 0x7fb37413fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56388/0 (socket says 172.21.15.62:56388) 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb373fff640 1 -- 172.21.15.62:0/4140881928 learned_addr learned my addr 172.21.15.62:0/4140881928 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb3737fe640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb35c003a10 0x7fb3741d6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb372ffd640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb374149170 0x7fb3741e43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.483 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb373fff640 1 -- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb35c003a10 msgr2=0x7fb3741d6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb373fff640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb35c003a10 0x7fb3741d6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb373fff640 1 -- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb374149170 msgr2=0x7fb3741e43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb373fff640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb374149170 0x7fb3741e43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb373fff640 1 -- 172.21.15.62:0/4140881928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb374149f20 con 0x7fb3741e4910 2024-04-18T10:06:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb372ffd640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb374149170 0x7fb3741e43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb3737fe640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb35c003a10 0x7fb3741d6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb373fff640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3741e4910 0x7fb37413fc70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb3740b76e0 tx=0x7fb36c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb35bfff640 1 -- 172.21.15.62:0/4140881928 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c00eb90 con 0x7fb3741e4910 2024-04-18T10:06:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb35bfff640 1 -- 172.21.15.62:0/4140881928 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb36c00ed30 con 0x7fb3741e4910 2024-04-18T10:06:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 -- 172.21.15.62:0/4140881928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb37414a200 con 0x7fb3741e4910 2024-04-18T10:06:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb35bfff640 1 -- 172.21.15.62:0/4140881928 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c014630 con 0x7fb3741e4910 2024-04-18T10:06:42.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 -- 172.21.15.62:0/4140881928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb374143ae0 con 0x7fb3741e4910 2024-04-18T10:06:42.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.481+0000 7fb379b56640 1 -- 172.21.15.62:0/4140881928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb374073bf0 con 0x7fb3741e4910 2024-04-18T10:06:42.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.485+0000 7fb35bfff640 1 -- 172.21.15.62:0/4140881928 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb36c0147d0 con 0x7fb3741e4910 2024-04-18T10:06:42.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.485+0000 7fb35bfff640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb360068380 0x7fb36006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.485+0000 7fb35bfff640 1 -- 172.21.15.62:0/4140881928 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fb36c086680 con 0x7fb3741e4910 2024-04-18T10:06:42.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.485+0000 7fb3737fe640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb360068380 0x7fb36006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.485+0000 7fb3737fe640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb360068380 0x7fb36006a840 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb364001750 tx=0x7fb3640011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:42.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.489+0000 7fb35bfff640 1 -- 172.21.15.62:0/4140881928 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb36c056cd0 con 0x7fb3741e4910 2024-04-18T10:06:42.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.653+0000 7fb379b56640 1 -- 172.21.15.62:0/4140881928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb374068b90 con 0x7fb3741e4910 2024-04-18T10:06:42.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb35bfff640 1 -- 172.21.15.62:0/4140881928 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16657 (secure 0 0 0) 0x7fb36c0565d0 con 0x7fb3741e4910 2024-04-18T10:06:42.656 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:42.656 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":30,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:39.937405+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:06:36.802689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-18T10:06:37.269237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:42.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb379b56640 1 -- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb360068380 msgr2=0x7fb36006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb379b56640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb360068380 0x7fb36006a840 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb364001750 tx=0x7fb3640011a0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb379b56640 1 -- 172.21.15.62:0/4140881928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3741e4910 msgr2=0x7fb37413fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb379b56640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3741e4910 0x7fb37413fc70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb3740b76e0 tx=0x7fb36c00e530 comp rx=0 tx=0).stop 2024-04-18T10:06:42.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb379b56640 1 -- 172.21.15.62:0/4140881928 shutdown_connections 2024-04-18T10:06:42.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb379b56640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb360068380 0x7fb36006a840 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb379b56640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3741e4910 0x7fb37413fc70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb379b56640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb374149170 0x7fb3741e43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb379b56640 1 --2- 172.21.15.62:0/4140881928 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb35c003a10 0x7fb3741d6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb379b56640 1 -- 172.21.15.62:0/4140881928 >> 172.21.15.62:0/4140881928 conn(0x7fb3740bd5c0 msgr2=0x7fb37414b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:42.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb379b56640 1 -- 172.21.15.62:0/4140881928 shutdown_connections 2024-04-18T10:06:42.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.657+0000 7fb379b56640 1 -- 172.21.15.62:0/4140881928 wait complete. 2024-04-18T10:06:42.677 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-18T10:06:42.677 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T10:06:42.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.793+0000 7f3889e8c640 1 Processor -- start 2024-04-18T10:06:42.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.793+0000 7f3889e8c640 1 -- start start 2024-04-18T10:06:42.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.793+0000 7f3889e8c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3884145f20 0x7f38841514f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.793+0000 7f3889e8c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f388406e030 con 0x7f38840b9d90 2024-04-18T10:06:42.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.793+0000 7f3889e8c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38840ba170 con 0x7f388409be30 2024-04-18T10:06:42.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.793+0000 7f3889e8c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f388409cd30 con 0x7f3884145f20 2024-04-18T10:06:42.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.793+0000 7f38837fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f38840b9d90 0x7f388406d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42860/0 (socket says 172.21.15.62:42860) 2024-04-18T10:06:42.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.793+0000 7f38837fe640 1 -- 172.21.15.62:0/2414087640 learned_addr learned my addr 172.21.15.62:0/2414087640 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:42.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.793+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2107612186 0 0) 0x7f388406e030 con 0x7f38840b9d90 2024-04-18T10:06:42.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.793+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3870003530 con 0x7f38840b9d90 2024-04-18T10:06:42.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.793+0000 7f3883fff640 1 --2- 172.21.15.62:0/2414087640 >> v2:172.21.15.145:3300/0 conn(0x7f3884145f20 0x7f38841514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 348375681 0 0) 0x7f3870003530 con 0x7f38840b9d90 2024-04-18T10:06:42.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 >> v2:172.21.15.145:3300/0 conn(0x7f3884145f20 msgr2=0x7f38841514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:42.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 --2- 172.21.15.62:0/2414087640 >> v2:172.21.15.145:3300/0 conn(0x7f3884145f20 0x7f38841514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 >> v1:172.21.15.145:6789/0 conn(0x7f388409be30 legacy=0x7f388406d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388414a060 con 0x7f38840b9d90 2024-04-18T10:06:42.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f3874001680 con 0x7f38840b9d90 2024-04-18T10:06:42.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 >> v1:172.21.15.62:6789/0 conn(0x7f38840b9d90 legacy=0x7f388406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 --2- 172.21.15.62:0/2414087640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3870003a10 0x7f3870003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 --2- 172.21.15.62:0/2414087640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3870004810 0x7f3870004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 --2- 172.21.15.62:0/2414087640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38700052f0 0x7f387000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f388409cd30 con 0x7f3870003a10 2024-04-18T10:06:42.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f387000c1f0 con 0x7f38700052f0 2024-04-18T10:06:42.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f387000c3f0 con 0x7f3870004810 2024-04-18T10:06:42.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38837fe640 1 --2- 172.21.15.62:0/2414087640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3870003a10 0x7f3870003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f3883fff640 1 --2- 172.21.15.62:0/2414087640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38700052f0 0x7f387000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f3882ffd640 1 --2- 172.21.15.62:0/2414087640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3870004810 0x7f3870004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38837fe640 1 -- 172.21.15.62:0/2414087640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3870004810 msgr2=0x7f3870004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:42.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38837fe640 1 --2- 172.21.15.62:0/2414087640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3870004810 0x7f3870004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38837fe640 1 -- 172.21.15.62:0/2414087640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38700052f0 msgr2=0x7f387000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:42.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38837fe640 1 --2- 172.21.15.62:0/2414087640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38700052f0 0x7f387000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f3883fff640 1 -- 172.21.15.62:0/2414087640 reap_dead start 2024-04-18T10:06:42.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38837fe640 1 -- 172.21.15.62:0/2414087640 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3870003530 con 0x7f3870003a10 2024-04-18T10:06:42.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f3882ffd640 1 --2- 172.21.15.62:0/2414087640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3870004810 0x7f3870004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:42.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38837fe640 1 --2- 172.21.15.62:0/2414087640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3870003a10 0x7f3870003e30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f387400c110 tx=0x7f387401cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457b827ac979b4c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874007c50 con 0x7f3870003a10 2024-04-18T10:06:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38740a9070 con 0x7f3870003a10 2024-04-18T10:06:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.797+0000 7f38827fc640 1 -- 172.21.15.62:0/2414087640 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874005d80 con 0x7f3870003a10 2024-04-18T10:06:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 -- 172.21.15.62:0/2414087640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3870003a10 msgr2=0x7f3870003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 --2- 172.21.15.62:0/2414087640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3870003a10 0x7f3870003e30 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f387400c110 tx=0x7f387401cb80 comp rx=0 tx=0).stop 2024-04-18T10:06:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 -- 172.21.15.62:0/2414087640 shutdown_connections 2024-04-18T10:06:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 --2- 172.21.15.62:0/2414087640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3870003a10 0x7f3870003e30 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 -- 172.21.15.62:0/2414087640 >> 172.21.15.62:0/2414087640 conn(0x7f388406c0d0 msgr2=0x7f38840b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 -- 172.21.15.62:0/2414087640 shutdown_connections 2024-04-18T10:06:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 -- 172.21.15.62:0/2414087640 wait complete. 2024-04-18T10:06:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 Processor -- start 2024-04-18T10:06:42.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 -- start start 2024-04-18T10:06:42.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3870003a10 0x7f38840b1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.803 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38840b2b10 0x7f38841e15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38841e1b10 0x7f3884142180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f38840b3e20 con 0x7f38840b2b10 2024-04-18T10:06:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f3884146330 con 0x7f38841e1b10 2024-04-18T10:06:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f38840b3fa0 con 0x7f3870003a10 2024-04-18T10:06:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3882ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38840b2b10 0x7f38841e15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3882ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38840b2b10 0x7f38841e15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56418/0 (socket says 172.21.15.62:56418) 2024-04-18T10:06:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3882ffd640 1 -- 172.21.15.62:0/1492860949 learned_addr learned my addr 172.21.15.62:0/1492860949 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f38837fe640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3870003a10 0x7f38840b1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3883fff640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38841e1b10 0x7f3884142180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3882ffd640 1 -- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3870003a10 msgr2=0x7f38840b1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3882ffd640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3870003a10 0x7f38840b1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.804 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3882ffd640 1 -- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38841e1b10 msgr2=0x7f3884142180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3882ffd640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38841e1b10 0x7f3884142180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3882ffd640 1 -- 172.21.15.62:0/1492860949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38840b0d00 con 0x7f38840b2b10 2024-04-18T10:06:42.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3882ffd640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38840b2b10 0x7f38841e15d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f386c00d4a0 tx=0x7f386c00d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:42.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f387bfff640 1 -- 172.21.15.62:0/1492860949 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c005280 con 0x7f38840b2b10 2024-04-18T10:06:42.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f387bfff640 1 -- 172.21.15.62:0/1492860949 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f386c005440 con 0x7f38840b2b10 2024-04-18T10:06:42.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 -- 172.21.15.62:0/1492860949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38841e1f90 con 0x7f38840b2b10 2024-04-18T10:06:42.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f387bfff640 1 -- 172.21.15.62:0/1492860949 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f386c007260 con 0x7f38840b2b10 2024-04-18T10:06:42.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 -- 172.21.15.62:0/1492860949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38840b0f80 con 0x7f38840b2b10 2024-04-18T10:06:42.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.801+0000 7f3889e8c640 1 -- 172.21.15.62:0/1492860949 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3884068b90 con 0x7f38840b2b10 2024-04-18T10:06:42.805 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.805+0000 7f387bfff640 1 -- 172.21.15.62:0/1492860949 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f386c019020 con 0x7f38840b2b10 2024-04-18T10:06:42.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.805+0000 7f387bfff640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f38640683b0 0x7f386406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:42.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.805+0000 7f387bfff640 1 -- 172.21.15.62:0/1492860949 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f386c085fe0 con 0x7f38840b2b10 2024-04-18T10:06:42.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.805+0000 7f38837fe640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f38640683b0 0x7f386406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:42.806 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.805+0000 7f38837fe640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f38640683b0 0x7f386406a870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f38740ab000 tx=0x7f38740b1040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:42.809 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.809+0000 7f387bfff640 1 -- 172.21.15.62:0/1492860949 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f38841e1f90 con 0x7f38840b2b10 2024-04-18T10:06:42.896 INFO:tasks.ceph.mds.a.smithi062.stderr:2024-04-18T10:06:42.897+0000 7f274e1b5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-18T10:06:42.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.965+0000 7f3889e8c640 1 -- 172.21.15.62:0/1492860949 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f38840aefc0 con 0x7f38640683b0 2024-04-18T10:06:42.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f387bfff640 1 -- 172.21.15.62:0/1492860949 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+237199 (secure 0 0 0) 0x7f38740f6ea0 con 0x7f38640683b0 2024-04-18T10:06:42.986 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:42.988 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-18T10:06:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f3889e8c640 1 -- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f38640683b0 msgr2=0x7f386406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f3889e8c640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f38640683b0 0x7f386406a870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f38740ab000 tx=0x7f38740b1040 comp rx=0 tx=0).stop 2024-04-18T10:06:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f3889e8c640 1 -- 172.21.15.62:0/1492860949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38840b2b10 msgr2=0x7f38841e15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f3889e8c640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38840b2b10 0x7f38841e15d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f386c00d4a0 tx=0x7f386c00d960 comp rx=0 tx=0).stop 2024-04-18T10:06:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f3889e8c640 1 -- 172.21.15.62:0/1492860949 shutdown_connections 2024-04-18T10:06:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f3889e8c640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f38640683b0 0x7f386406a870 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f3889e8c640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38841e1b10 0x7f3884142180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f3889e8c640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38840b2b10 0x7f38841e15d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f3889e8c640 1 --2- 172.21.15.62:0/1492860949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3870003a10 0x7f38840b1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:42.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f3889e8c640 1 -- 172.21.15.62:0/1492860949 >> 172.21.15.62:0/1492860949 conn(0x7f388406c0d0 msgr2=0x7f3884144530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:42.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f3889e8c640 1 -- 172.21.15.62:0/1492860949 shutdown_connections 2024-04-18T10:06:42.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:42.985+0000 7f3889e8c640 1 -- 172.21.15.62:0/1492860949 wait complete. 2024-04-18T10:06:43.002 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2024-04-18T10:06:41.139234+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":219604,"kb_used_data":3400,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755116,"statfs":{"total":773094113280,"available":772869238784,"internally_reserved":0,"allocated":3481600,"data_stored":2008928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.210557"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:37.925903+0000","last_change":"2024-04-18T10:06:37.925903+0000","last_active":"2024-04-18T10:06:37.925903+0000","last_peered":"2024-04-18T10:06:37.925903+0000","last_clean":"2024-04-18T10:06:37.925903+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:37.925903+0000","last_undegraded":"2024-04-18T10:06:37.925903+0000","last_fullsized":"2024-04-18T10:06:37.925903+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-18T10:06:36.915753+0000","last_change":"2024-04-18T10:06:36.915753+0000","last_active":"2024-04-18T10:06:36.915753+0000","last_peered":"2024-04-18T10:06:36.915753+0000","last_clean":"2024-04-18T10:06:36.915753+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-18T10:06:36.915753+0000","last_undegraded":"2024-04-18T10:06:36.915753+0000","last_fullsized":"2024-04-18T10:06:36.915753+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-18T10:06:33.786279+0000","last_change":"2024-04-18T10:05:49.183850+0000","last_active":"2024-04-18T10:06:33.786279+0000","last_peered":"2024-04-18T10:06:33.786279+0000","last_clean":"2024-04-18T10:06:33.786279+0000","last_became_active":"2024-04-18T10:05:49.181471+0000","last_became_peered":"2024-04-18T10:05:49.181471+0000","last_unstale":"2024-04-18T10:06:33.786279+0000","last_undegraded":"2024-04-18T10:06:33.786279+0000","last_fullsized":"2024-04-18T10:06:33.786279+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_clean_scrub_stamp":"2024-04-18T10:05:47.511486+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:06:51.214547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574863,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":693892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":103524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":103524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":103524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":103524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":286720,"data_stored":103524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":103524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574861,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":880640,"data_stored":693892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T10:06:43.006 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T10:06:43.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f44891e6640 1 Processor -- start 2024-04-18T10:06:43.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f44891e6640 1 -- start start 2024-04-18T10:06:43.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f44891e6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f448409be30 0x7f4484145f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f44891e6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f448406d920 0x7f44841504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f44891e6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44840ba170 con 0x7f44840b9d90 2024-04-18T10:06:43.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f44891e6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f448409cd30 con 0x7f448409be30 2024-04-18T10:06:43.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f44891e6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f4484146460 con 0x7f448406d920 2024-04-18T10:06:43.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4482d9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f44840b9d90 0x7f448406d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42876/0 (socket says 172.21.15.62:42876) 2024-04-18T10:06:43.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4482d9d640 1 -- 172.21.15.62:0/316982006 learned_addr learned my addr 172.21.15.62:0/316982006 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f448259c640 1 --2- 172.21.15.62:0/316982006 >> v2:172.21.15.145:3300/0 conn(0x7f448409be30 0x7f4484145f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f448359e640 1 --2- 172.21.15.62:0/316982006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f448406d920 0x7f44841504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4142947798 0 0) 0x7f44840ba170 con 0x7f44840b9d90 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f446c003530 con 0x7f44840b9d90 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 425899294 0 0) 0x7f446c003530 con 0x7f44840b9d90 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f448406d920 msgr2=0x7f44841504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 --2- 172.21.15.62:0/316982006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f448406d920 0x7f44841504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 >> v2:172.21.15.145:3300/0 conn(0x7f448409be30 msgr2=0x7f4484145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 --2- 172.21.15.62:0/316982006 >> v2:172.21.15.145:3300/0 conn(0x7f448409be30 0x7f4484145f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f448414a060 con 0x7f44840b9d90 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f4478001680 con 0x7f44840b9d90 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 >> v1:172.21.15.62:6789/0 conn(0x7f44840b9d90 legacy=0x7f448406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 --2- 172.21.15.62:0/316982006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f446c0039c0 0x7f446c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 --2- 172.21.15.62:0/316982006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f448406d920 0x7f446c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 --2- 172.21.15.62:0/316982006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f446c004d50 0x7f446c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f448409cd30 con 0x7f446c0039c0 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f4484146460 con 0x7f446c004d50 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f446c00ba50 con 0x7f448406d920 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4482d9d640 1 --2- 172.21.15.62:0/316982006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f446c0039c0 0x7f446c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f448359e640 1 --2- 172.21.15.62:0/316982006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f446c004d50 0x7f446c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4482d9d640 1 -- 172.21.15.62:0/316982006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f448406d920 msgr2=0x7f446c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4482d9d640 1 --2- 172.21.15.62:0/316982006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f448406d920 0x7f446c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4482d9d640 1 -- 172.21.15.62:0/316982006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f446c004d50 msgr2=0x7f446c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4482d9d640 1 --2- 172.21.15.62:0/316982006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f446c004d50 0x7f446c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f4482d9d640 1 -- 172.21.15.62:0/316982006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446c003530 con 0x7f446c0039c0 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.165+0000 7f448359e640 1 --2- 172.21.15.62:0/316982006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f446c004d50 0x7f446c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f4482d9d640 1 --2- 172.21.15.62:0/316982006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f446c0039c0 0x7f446c003de0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f447800c2c0 tx=0x7f447801cd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13110c93bc5db3fe server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:43.172 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4478007d60 con 0x7f446c0039c0 2024-04-18T10:06:43.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44780a9020 con 0x7f446c0039c0 2024-04-18T10:06:43.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f4481d9b640 1 -- 172.21.15.62:0/316982006 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4478005ae0 con 0x7f446c0039c0 2024-04-18T10:06:43.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 -- 172.21.15.62:0/316982006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f446c0039c0 msgr2=0x7f446c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 --2- 172.21.15.62:0/316982006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f446c0039c0 0x7f446c003de0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f447800c2c0 tx=0x7f447801cd40 comp rx=0 tx=0).stop 2024-04-18T10:06:43.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f448359e640 1 -- 172.21.15.62:0/316982006 reap_dead start 2024-04-18T10:06:43.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 -- 172.21.15.62:0/316982006 shutdown_connections 2024-04-18T10:06:43.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 -- 172.21.15.62:0/316982006 >> 172.21.15.62:0/316982006 conn(0x7f448406c0d0 msgr2=0x7f44840b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:43.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 -- 172.21.15.62:0/316982006 shutdown_connections 2024-04-18T10:06:43.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 -- 172.21.15.62:0/316982006 wait complete. 2024-04-18T10:06:43.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 Processor -- start 2024-04-18T10:06:43.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 -- start start 2024-04-18T10:06:43.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4484143a40 0x7f44841e1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44841e2160 0x7f4484140580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4484140ac0 0x7f44840b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f44840b3fa0 con 0x7f44841e2160 2024-04-18T10:06:43.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f44840b3e20 con 0x7f4484143a40 2024-04-18T10:06:43.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f4484150cf0 con 0x7f4484140ac0 2024-04-18T10:06:43.176 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f448259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44841e2160 0x7f4484140580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f448259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44841e2160 0x7f4484140580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56426/0 (socket says 172.21.15.62:56426) 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f448259c640 1 -- 172.21.15.62:0/2711084958 learned_addr learned my addr 172.21.15.62:0/2711084958 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f4482d9d640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4484143a40 0x7f44841e1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f448259c640 1 -- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4484140ac0 msgr2=0x7f44840b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f448259c640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4484140ac0 0x7f44840b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f448259c640 1 -- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4484143a40 msgr2=0x7f44841e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f448259c640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4484143a40 0x7f44841e1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f448259c640 1 -- 172.21.15.62:0/2711084958 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44840717b0 con 0x7f44841e2160 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f448259c640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44841e2160 0x7f4484140580 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f447c009de0 tx=0x7f447c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f446b7fe640 1 -- 172.21.15.62:0/2711084958 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c018020 con 0x7f44841e2160 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f446b7fe640 1 -- 172.21.15.62:0/2711084958 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f447c00ce50 con 0x7f44841e2160 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 -- 172.21.15.62:0/2711084958 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4484070910 con 0x7f44841e2160 2024-04-18T10:06:43.177 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f446b7fe640 1 -- 172.21.15.62:0/2711084958 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c0146c0 con 0x7f44841e2160 2024-04-18T10:06:43.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.169+0000 7f44891e6640 1 -- 172.21.15.62:0/2711084958 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4484071a20 con 0x7f44841e2160 2024-04-18T10:06:43.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.173+0000 7f44891e6640 1 -- 172.21.15.62:0/2711084958 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4484068b90 con 0x7f44841e2160 2024-04-18T10:06:43.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.173+0000 7f446b7fe640 1 -- 172.21.15.62:0/2711084958 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f447c01b020 con 0x7f44841e2160 2024-04-18T10:06:43.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.173+0000 7f446b7fe640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f44700683b0 0x7f447006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.173+0000 7f446b7fe640 1 -- 172.21.15.62:0/2711084958 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f447c0875b0 con 0x7f44841e2160 2024-04-18T10:06:43.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.173+0000 7f4482d9d640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f44700683b0 0x7f447006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.173+0000 7f4482d9d640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f44700683b0 0x7f447006a870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f44780aa000 tx=0x7f4478001a50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:43.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.177+0000 7f446b7fe640 1 -- 172.21.15.62:0/2711084958 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f447c022020 con 0x7f44841e2160 2024-04-18T10:06:43.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.333+0000 7f44891e6640 1 -- 172.21.15.62:0/2711084958 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4484144550 con 0x7f44700683b0 2024-04-18T10:06:43.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.349+0000 7f446b7fe640 1 -- 172.21.15.62:0/2711084958 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251976 (secure 0 0 0) 0x7f4484144550 con 0x7f44700683b0 2024-04-18T10:06:43.353 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:43.354 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-18T10:06:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.353+0000 7f44891e6640 1 -- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f44700683b0 msgr2=0x7f447006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.353+0000 7f44891e6640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f44700683b0 0x7f447006a870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f44780aa000 tx=0x7f4478001a50 comp rx=0 tx=0).stop 2024-04-18T10:06:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.353+0000 7f44891e6640 1 -- 172.21.15.62:0/2711084958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44841e2160 msgr2=0x7f4484140580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.353+0000 7f44891e6640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44841e2160 0x7f4484140580 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f447c009de0 tx=0x7f447c00c490 comp rx=0 tx=0).stop 2024-04-18T10:06:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.353+0000 7f44891e6640 1 -- 172.21.15.62:0/2711084958 shutdown_connections 2024-04-18T10:06:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.353+0000 7f44891e6640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f44700683b0 0x7f447006a870 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.353+0000 7f44891e6640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4484140ac0 0x7f44840b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.353+0000 7f44891e6640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44841e2160 0x7f4484140580 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.353+0000 7f44891e6640 1 --2- 172.21.15.62:0/2711084958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4484143a40 0x7f44841e1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.353+0000 7f44891e6640 1 -- 172.21.15.62:0/2711084958 >> 172.21.15.62:0/2711084958 conn(0x7f448406c0d0 msgr2=0x7f4484074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:43.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.353+0000 7f44891e6640 1 -- 172.21.15.62:0/2711084958 shutdown_connections 2024-04-18T10:06:43.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.353+0000 7f44891e6640 1 -- 172.21.15.62:0/2711084958 wait complete. 2024-04-18T10:06:43.370 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-04-18T10:06:43.139848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220064,"kb_used_data":3848,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754656,"statfs":{"total":773094113280,"available":772868767744,"internally_reserved":0,"allocated":3940352,"data_stored":2215020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001759"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185453+0000","last_change":"2024-04-18T10:06:38.998684+0000","last_active":"2024-04-18T10:06:40.185453+0000","last_peered":"2024-04-18T10:06:40.185453+0000","last_clean":"2024-04-18T10:06:40.185453+0000","last_became_active":"2024-04-18T10:06:38.997878+0000","last_became_peered":"2024-04-18T10:06:38.997878+0000","last_unstale":"2024-04-18T10:06:40.185453+0000","last_undegraded":"2024-04-18T10:06:40.185453+0000","last_fullsized":"2024-04-18T10:06:40.185453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:28:46.488660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186674+0000","last_change":"2024-04-18T10:06:38.962074+0000","last_active":"2024-04-18T10:06:40.186674+0000","last_peered":"2024-04-18T10:06:40.186674+0000","last_clean":"2024-04-18T10:06:40.186674+0000","last_became_active":"2024-04-18T10:06:38.961784+0000","last_became_peered":"2024-04-18T10:06:38.961784+0000","last_unstale":"2024-04-18T10:06:40.186674+0000","last_undegraded":"2024-04-18T10:06:40.186674+0000","last_fullsized":"2024-04-18T10:06:40.186674+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:02:04.700746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947698+0000","last_change":"2024-04-18T10:06:38.975309+0000","last_active":"2024-04-18T10:06:39.947698+0000","last_peered":"2024-04-18T10:06:39.947698+0000","last_clean":"2024-04-18T10:06:39.947698+0000","last_became_active":"2024-04-18T10:06:38.974604+0000","last_became_peered":"2024-04-18T10:06:38.974604+0000","last_unstale":"2024-04-18T10:06:39.947698+0000","last_undegraded":"2024-04-18T10:06:39.947698+0000","last_fullsized":"2024-04-18T10:06:39.947698+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:22:07.145450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948118+0000","last_change":"2024-04-18T10:06:38.982034+0000","last_active":"2024-04-18T10:06:39.948118+0000","last_peered":"2024-04-18T10:06:39.948118+0000","last_clean":"2024-04-18T10:06:39.948118+0000","last_became_active":"2024-04-18T10:06:38.981708+0000","last_became_peered":"2024-04-18T10:06:38.981708+0000","last_unstale":"2024-04-18T10:06:39.948118+0000","last_undegraded":"2024-04-18T10:06:39.948118+0000","last_fullsized":"2024-04-18T10:06:39.948118+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:28:23.502161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.183652+0000","last_change":"2024-04-18T10:06:38.962121+0000","last_active":"2024-04-18T10:06:40.183652+0000","last_peered":"2024-04-18T10:06:40.183652+0000","last_clean":"2024-04-18T10:06:40.183652+0000","last_became_active":"2024-04-18T10:06:38.961828+0000","last_became_peered":"2024-04-18T10:06:38.961828+0000","last_unstale":"2024-04-18T10:06:40.183652+0000","last_undegraded":"2024-04-18T10:06:40.183652+0000","last_fullsized":"2024-04-18T10:06:40.183652+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:08:12.887106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185494+0000","last_change":"2024-04-18T10:06:38.996916+0000","last_active":"2024-04-18T10:06:40.185494+0000","last_peered":"2024-04-18T10:06:40.185494+0000","last_clean":"2024-04-18T10:06:40.185494+0000","last_became_active":"2024-04-18T10:06:38.996524+0000","last_became_peered":"2024-04-18T10:06:38.996524+0000","last_unstale":"2024-04-18T10:06:40.185494+0000","last_undegraded":"2024-04-18T10:06:40.185494+0000","last_fullsized":"2024-04-18T10:06:40.185494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:36:59.162611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186255+0000","last_change":"2024-04-18T10:06:39.001331+0000","last_active":"2024-04-18T10:06:40.186255+0000","last_peered":"2024-04-18T10:06:40.186255+0000","last_clean":"2024-04-18T10:06:40.186255+0000","last_became_active":"2024-04-18T10:06:39.000672+0000","last_became_peered":"2024-04-18T10:06:39.000672+0000","last_unstale":"2024-04-18T10:06:40.186255+0000","last_undegraded":"2024-04-18T10:06:40.186255+0000","last_fullsized":"2024-04-18T10:06:40.186255+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:15:08.527241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.184617+0000","last_change":"2024-04-18T10:06:38.985387+0000","last_active":"2024-04-18T10:06:40.184617+0000","last_peered":"2024-04-18T10:06:40.184617+0000","last_clean":"2024-04-18T10:06:40.184617+0000","last_became_active":"2024-04-18T10:06:38.984432+0000","last_became_peered":"2024-04-18T10:06:38.984432+0000","last_unstale":"2024-04-18T10:06:40.184617+0000","last_undegraded":"2024-04-18T10:06:40.184617+0000","last_fullsized":"2024-04-18T10:06:40.184617+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:12:32.789737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948988+0000","last_change":"2024-04-18T10:06:38.976966+0000","last_active":"2024-04-18T10:06:39.948988+0000","last_peered":"2024-04-18T10:06:39.948988+0000","last_clean":"2024-04-18T10:06:39.948988+0000","last_became_active":"2024-04-18T10:06:38.974251+0000","last_became_peered":"2024-04-18T10:06:38.974251+0000","last_unstale":"2024-04-18T10:06:39.948988+0000","last_undegraded":"2024-04-18T10:06:39.948988+0000","last_fullsized":"2024-04-18T10:06:39.948988+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:36:59.162611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.949321+0000","last_change":"2024-04-18T10:06:38.971322+0000","last_active":"2024-04-18T10:06:39.949321+0000","last_peered":"2024-04-18T10:06:39.949321+0000","last_clean":"2024-04-18T10:06:39.949321+0000","last_became_active":"2024-04-18T10:06:38.970785+0000","last_became_peered":"2024-04-18T10:06:38.970785+0000","last_unstale":"2024-04-18T10:06:39.949321+0000","last_undegraded":"2024-04-18T10:06:39.949321+0000","last_fullsized":"2024-04-18T10:06:39.949321+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:26:33.642993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186995+0000","last_change":"2024-04-18T10:06:38.998488+0000","last_active":"2024-04-18T10:06:40.186995+0000","last_peered":"2024-04-18T10:06:40.186995+0000","last_clean":"2024-04-18T10:06:40.186995+0000","last_became_active":"2024-04-18T10:06:38.995263+0000","last_became_peered":"2024-04-18T10:06:38.995263+0000","last_unstale":"2024-04-18T10:06:40.186995+0000","last_undegraded":"2024-04-18T10:06:40.186995+0000","last_fullsized":"2024-04-18T10:06:40.186995+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:02:04.700746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.946909+0000","last_change":"2024-04-18T10:06:38.975607+0000","last_active":"2024-04-18T10:06:39.946909+0000","last_peered":"2024-04-18T10:06:39.946909+0000","last_clean":"2024-04-18T10:06:39.946909+0000","last_became_active":"2024-04-18T10:06:38.974834+0000","last_became_peered":"2024-04-18T10:06:38.974834+0000","last_unstale":"2024-04-18T10:06:39.946909+0000","last_undegraded":"2024-04-18T10:06:39.946909+0000","last_fullsized":"2024-04-18T10:06:39.946909+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:55:16.901514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.183522+0000","last_change":"2024-04-18T10:06:38.972583+0000","last_active":"2024-04-18T10:06:40.183522+0000","last_peered":"2024-04-18T10:06:40.183522+0000","last_clean":"2024-04-18T10:06:40.183522+0000","last_became_active":"2024-04-18T10:06:38.972140+0000","last_became_peered":"2024-04-18T10:06:38.972140+0000","last_unstale":"2024-04-18T10:06:40.183522+0000","last_undegraded":"2024-04-18T10:06:40.183522+0000","last_fullsized":"2024-04-18T10:06:40.183522+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:34:48.959180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185820+0000","last_change":"2024-04-18T10:06:38.961590+0000","last_active":"2024-04-18T10:06:40.185820+0000","last_peered":"2024-04-18T10:06:40.185820+0000","last_clean":"2024-04-18T10:06:40.185820+0000","last_became_active":"2024-04-18T10:06:38.956923+0000","last_became_peered":"2024-04-18T10:06:38.956923+0000","last_unstale":"2024-04-18T10:06:40.185820+0000","last_undegraded":"2024-04-18T10:06:40.185820+0000","last_fullsized":"2024-04-18T10:06:40.185820+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:15:08.527241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.945849+0000","last_change":"2024-04-18T10:06:38.987052+0000","last_active":"2024-04-18T10:06:39.945849+0000","last_peered":"2024-04-18T10:06:39.945849+0000","last_clean":"2024-04-18T10:06:39.945849+0000","last_became_active":"2024-04-18T10:06:38.986284+0000","last_became_peered":"2024-04-18T10:06:38.986284+0000","last_unstale":"2024-04-18T10:06:39.945849+0000","last_undegraded":"2024-04-18T10:06:39.945849+0000","last_fullsized":"2024-04-18T10:06:39.945849+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:36:59.162611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185213+0000","last_change":"2024-04-18T10:06:38.994229+0000","last_active":"2024-04-18T10:06:40.185213+0000","last_peered":"2024-04-18T10:06:40.185213+0000","last_clean":"2024-04-18T10:06:40.185213+0000","last_became_active":"2024-04-18T10:06:38.993214+0000","last_became_peered":"2024-04-18T10:06:38.993214+0000","last_unstale":"2024-04-18T10:06:40.185213+0000","last_undegraded":"2024-04-18T10:06:40.185213+0000","last_fullsized":"2024-04-18T10:06:40.185213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:51:17.460536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186910+0000","last_change":"2024-04-18T10:06:38.993396+0000","last_active":"2024-04-18T10:06:40.186910+0000","last_peered":"2024-04-18T10:06:40.186910+0000","last_clean":"2024-04-18T10:06:40.186910+0000","last_became_active":"2024-04-18T10:06:38.992693+0000","last_became_peered":"2024-04-18T10:06:38.992693+0000","last_unstale":"2024-04-18T10:06:40.186910+0000","last_undegraded":"2024-04-18T10:06:40.186910+0000","last_fullsized":"2024-04-18T10:06:40.186910+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:47:38.632426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.187034+0000","last_change":"2024-04-18T10:06:38.974822+0000","last_active":"2024-04-18T10:06:40.187034+0000","last_peered":"2024-04-18T10:06:40.187034+0000","last_clean":"2024-04-18T10:06:40.187034+0000","last_became_active":"2024-04-18T10:06:38.974415+0000","last_became_peered":"2024-04-18T10:06:38.974415+0000","last_unstale":"2024-04-18T10:06:40.187034+0000","last_undegraded":"2024-04-18T10:06:40.187034+0000","last_fullsized":"2024-04-18T10:06:40.187034+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:22:07.145450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186202+0000","last_change":"2024-04-18T10:06:38.975659+0000","last_active":"2024-04-18T10:06:40.186202+0000","last_peered":"2024-04-18T10:06:40.186202+0000","last_clean":"2024-04-18T10:06:40.186202+0000","last_became_active":"2024-04-18T10:06:38.974687+0000","last_became_peered":"2024-04-18T10:06:38.974687+0000","last_unstale":"2024-04-18T10:06:40.186202+0000","last_undegraded":"2024-04-18T10:06:40.186202+0000","last_fullsized":"2024-04-18T10:06:40.186202+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:53:56.112173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.184069+0000","last_change":"2024-04-18T10:06:38.996796+0000","last_active":"2024-04-18T10:06:40.184069+0000","last_peered":"2024-04-18T10:06:40.184069+0000","last_clean":"2024-04-18T10:06:40.184069+0000","last_became_active":"2024-04-18T10:06:38.996480+0000","last_became_peered":"2024-04-18T10:06:38.996480+0000","last_unstale":"2024-04-18T10:06:40.184069+0000","last_undegraded":"2024-04-18T10:06:40.184069+0000","last_fullsized":"2024-04-18T10:06:40.184069+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:22:32.972316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947021+0000","last_change":"2024-04-18T10:06:38.972200+0000","last_active":"2024-04-18T10:06:39.947021+0000","last_peered":"2024-04-18T10:06:39.947021+0000","last_clean":"2024-04-18T10:06:39.947021+0000","last_became_active":"2024-04-18T10:06:38.971353+0000","last_became_peered":"2024-04-18T10:06:38.971353+0000","last_unstale":"2024-04-18T10:06:39.947021+0000","last_undegraded":"2024-04-18T10:06:39.947021+0000","last_fullsized":"2024-04-18T10:06:39.947021+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:34:42.479122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947347+0000","last_change":"2024-04-18T10:06:38.973121+0000","last_active":"2024-04-18T10:06:39.947347+0000","last_peered":"2024-04-18T10:06:39.947347+0000","last_clean":"2024-04-18T10:06:39.947347+0000","last_became_active":"2024-04-18T10:06:38.972399+0000","last_became_peered":"2024-04-18T10:06:38.972399+0000","last_unstale":"2024-04-18T10:06:39.947347+0000","last_undegraded":"2024-04-18T10:06:39.947347+0000","last_fullsized":"2024-04-18T10:06:39.947347+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:26:33.642993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.943566+0000","last_change":"2024-04-18T10:06:38.984684+0000","last_active":"2024-04-18T10:06:39.943566+0000","last_peered":"2024-04-18T10:06:39.943566+0000","last_clean":"2024-04-18T10:06:39.943566+0000","last_became_active":"2024-04-18T10:06:38.984440+0000","last_became_peered":"2024-04-18T10:06:38.984440+0000","last_unstale":"2024-04-18T10:06:39.943566+0000","last_undegraded":"2024-04-18T10:06:39.943566+0000","last_fullsized":"2024-04-18T10:06:39.943566+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:17:06.354662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.945399+0000","last_change":"2024-04-18T10:06:38.974363+0000","last_active":"2024-04-18T10:06:39.945399+0000","last_peered":"2024-04-18T10:06:39.945399+0000","last_clean":"2024-04-18T10:06:39.945399+0000","last_became_active":"2024-04-18T10:06:38.954076+0000","last_became_peered":"2024-04-18T10:06:38.954076+0000","last_unstale":"2024-04-18T10:06:39.945399+0000","last_undegraded":"2024-04-18T10:06:39.945399+0000","last_fullsized":"2024-04-18T10:06:39.945399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:22:50.350533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948921+0000","last_change":"2024-04-18T10:06:38.997887+0000","last_active":"2024-04-18T10:06:39.948921+0000","last_peered":"2024-04-18T10:06:39.948921+0000","last_clean":"2024-04-18T10:06:39.948921+0000","last_became_active":"2024-04-18T10:06:38.997593+0000","last_became_peered":"2024-04-18T10:06:38.997593+0000","last_unstale":"2024-04-18T10:06:39.948921+0000","last_undegraded":"2024-04-18T10:06:39.948921+0000","last_fullsized":"2024-04-18T10:06:39.948921+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:12:32.789737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.187442+0000","last_change":"2024-04-18T10:06:38.998105+0000","last_active":"2024-04-18T10:06:40.187442+0000","last_peered":"2024-04-18T10:06:40.187442+0000","last_clean":"2024-04-18T10:06:40.187442+0000","last_became_active":"2024-04-18T10:06:38.997818+0000","last_became_peered":"2024-04-18T10:06:38.997818+0000","last_unstale":"2024-04-18T10:06:40.187442+0000","last_undegraded":"2024-04-18T10:06:40.187442+0000","last_fullsized":"2024-04-18T10:06:40.187442+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T13:28:49.144196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948673+0000","last_change":"2024-04-18T10:06:38.972925+0000","last_active":"2024-04-18T10:06:39.948673+0000","last_peered":"2024-04-18T10:06:39.948673+0000","last_clean":"2024-04-18T10:06:39.948673+0000","last_became_active":"2024-04-18T10:06:38.970845+0000","last_became_peered":"2024-04-18T10:06:38.970845+0000","last_unstale":"2024-04-18T10:06:39.948673+0000","last_undegraded":"2024-04-18T10:06:39.948673+0000","last_fullsized":"2024-04-18T10:06:39.948673+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:44:02.108887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.946317+0000","last_change":"2024-04-18T10:06:38.992647+0000","last_active":"2024-04-18T10:06:39.946317+0000","last_peered":"2024-04-18T10:06:39.946317+0000","last_clean":"2024-04-18T10:06:39.946317+0000","last_became_active":"2024-04-18T10:06:38.991768+0000","last_became_peered":"2024-04-18T10:06:38.991768+0000","last_unstale":"2024-04-18T10:06:39.946317+0000","last_undegraded":"2024-04-18T10:06:39.946317+0000","last_fullsized":"2024-04-18T10:06:39.946317+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:15:08.527241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947299+0000","last_change":"2024-04-18T10:06:38.969712+0000","last_active":"2024-04-18T10:06:39.947299+0000","last_peered":"2024-04-18T10:06:39.947299+0000","last_clean":"2024-04-18T10:06:39.947299+0000","last_became_active":"2024-04-18T10:06:38.969036+0000","last_became_peered":"2024-04-18T10:06:38.969036+0000","last_unstale":"2024-04-18T10:06:39.947299+0000","last_undegraded":"2024-04-18T10:06:39.947299+0000","last_fullsized":"2024-04-18T10:06:39.947299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:12:32.789737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186268+0000","last_change":"2024-04-18T10:06:38.998597+0000","last_active":"2024-04-18T10:06:40.186268+0000","last_peered":"2024-04-18T10:06:40.186268+0000","last_clean":"2024-04-18T10:06:40.186268+0000","last_became_active":"2024-04-18T10:06:38.993689+0000","last_became_peered":"2024-04-18T10:06:38.993689+0000","last_unstale":"2024-04-18T10:06:40.186268+0000","last_undegraded":"2024-04-18T10:06:40.186268+0000","last_fullsized":"2024-04-18T10:06:40.186268+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:31:02.940425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186176+0000","last_change":"2024-04-18T10:06:38.964712+0000","last_active":"2024-04-18T10:06:40.186176+0000","last_peered":"2024-04-18T10:06:40.186176+0000","last_clean":"2024-04-18T10:06:40.186176+0000","last_became_active":"2024-04-18T10:06:38.963754+0000","last_became_peered":"2024-04-18T10:06:38.963754+0000","last_unstale":"2024-04-18T10:06:40.186176+0000","last_undegraded":"2024-04-18T10:06:40.186176+0000","last_fullsized":"2024-04-18T10:06:40.186176+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:17:06.354662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.184646+0000","last_change":"2024-04-18T10:06:38.994393+0000","last_active":"2024-04-18T10:06:40.184646+0000","last_peered":"2024-04-18T10:06:40.184646+0000","last_clean":"2024-04-18T10:06:40.184646+0000","last_became_active":"2024-04-18T10:06:38.993734+0000","last_became_peered":"2024-04-18T10:06:38.993734+0000","last_unstale":"2024-04-18T10:06:40.184646+0000","last_undegraded":"2024-04-18T10:06:40.184646+0000","last_fullsized":"2024-04-18T10:06:40.184646+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:26:33.642993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.944279+0000","last_change":"2024-04-18T10:06:38.997963+0000","last_active":"2024-04-18T10:06:39.944279+0000","last_peered":"2024-04-18T10:06:39.944279+0000","last_clean":"2024-04-18T10:06:39.944279+0000","last_became_active":"2024-04-18T10:06:38.997667+0000","last_became_peered":"2024-04-18T10:06:38.997667+0000","last_unstale":"2024-04-18T10:06:39.944279+0000","last_undegraded":"2024-04-18T10:06:39.944279+0000","last_fullsized":"2024-04-18T10:06:39.944279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:18:23.695897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"29'4","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185822+0000","last_change":"2024-04-18T10:06:37.978378+0000","last_active":"2024-04-18T10:06:40.185822+0000","last_peered":"2024-04-18T10:06:40.185822+0000","last_clean":"2024-04-18T10:06:40.185822+0000","last_became_active":"2024-04-18T10:06:37.977861+0000","last_became_peered":"2024-04-18T10:06:37.977861+0000","last_unstale":"2024-04-18T10:06:40.185822+0000","last_undegraded":"2024-04-18T10:06:40.185822+0000","last_fullsized":"2024-04-18T10:06:40.185822+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:24:02.490458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185878+0000","last_change":"2024-04-18T10:06:38.963949+0000","last_active":"2024-04-18T10:06:40.185878+0000","last_peered":"2024-04-18T10:06:40.185878+0000","last_clean":"2024-04-18T10:06:40.185878+0000","last_became_active":"2024-04-18T10:06:38.963103+0000","last_became_peered":"2024-04-18T10:06:38.963103+0000","last_unstale":"2024-04-18T10:06:40.185878+0000","last_undegraded":"2024-04-18T10:06:40.185878+0000","last_fullsized":"2024-04-18T10:06:40.185878+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:37:31.483243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.944560+0000","last_change":"2024-04-18T10:06:37.958325+0000","last_active":"2024-04-18T10:06:39.944560+0000","last_peered":"2024-04-18T10:06:39.944560+0000","last_clean":"2024-04-18T10:06:39.944560+0000","last_became_active":"2024-04-18T10:06:37.957675+0000","last_became_peered":"2024-04-18T10:06:37.957675+0000","last_unstale":"2024-04-18T10:06:39.944560+0000","last_undegraded":"2024-04-18T10:06:39.944560+0000","last_fullsized":"2024-04-18T10:06:39.944560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:47:47.427543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947957+0000","last_change":"2024-04-18T10:06:38.967105+0000","last_active":"2024-04-18T10:06:39.947957+0000","last_peered":"2024-04-18T10:06:39.947957+0000","last_clean":"2024-04-18T10:06:39.947957+0000","last_became_active":"2024-04-18T10:06:38.966054+0000","last_became_peered":"2024-04-18T10:06:38.966054+0000","last_unstale":"2024-04-18T10:06:39.947957+0000","last_undegraded":"2024-04-18T10:06:39.947957+0000","last_fullsized":"2024-04-18T10:06:39.947957+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:22:50.350533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186177+0000","last_change":"2024-04-18T10:06:37.966575+0000","last_active":"2024-04-18T10:06:40.186177+0000","last_peered":"2024-04-18T10:06:40.186177+0000","last_clean":"2024-04-18T10:06:40.186177+0000","last_became_active":"2024-04-18T10:06:37.966034+0000","last_became_peered":"2024-04-18T10:06:37.966034+0000","last_unstale":"2024-04-18T10:06:40.186177+0000","last_undegraded":"2024-04-18T10:06:40.186177+0000","last_fullsized":"2024-04-18T10:06:40.186177+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:28:45.478509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948746+0000","last_change":"2024-04-18T10:06:38.982201+0000","last_active":"2024-04-18T10:06:39.948746+0000","last_peered":"2024-04-18T10:06:39.948746+0000","last_clean":"2024-04-18T10:06:39.948746+0000","last_became_active":"2024-04-18T10:06:38.981811+0000","last_became_peered":"2024-04-18T10:06:38.981811+0000","last_unstale":"2024-04-18T10:06:39.948746+0000","last_undegraded":"2024-04-18T10:06:39.948746+0000","last_fullsized":"2024-04-18T10:06:39.948746+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:34:42.479122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.184063+0000","last_change":"2024-04-18T10:06:37.965547+0000","last_active":"2024-04-18T10:06:40.184063+0000","last_peered":"2024-04-18T10:06:40.184063+0000","last_clean":"2024-04-18T10:06:40.184063+0000","last_became_active":"2024-04-18T10:06:37.964945+0000","last_became_peered":"2024-04-18T10:06:37.964945+0000","last_unstale":"2024-04-18T10:06:40.184063+0000","last_undegraded":"2024-04-18T10:06:40.184063+0000","last_fullsized":"2024-04-18T10:06:40.184063+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:29:38.522460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186632+0000","last_change":"2024-04-18T10:06:38.993704+0000","last_active":"2024-04-18T10:06:40.186632+0000","last_peered":"2024-04-18T10:06:40.186632+0000","last_clean":"2024-04-18T10:06:40.186632+0000","last_became_active":"2024-04-18T10:06:38.993051+0000","last_became_peered":"2024-04-18T10:06:38.993051+0000","last_unstale":"2024-04-18T10:06:40.186632+0000","last_undegraded":"2024-04-18T10:06:40.186632+0000","last_fullsized":"2024-04-18T10:06:40.186632+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:53:56.112173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947596+0000","last_change":"2024-04-18T10:06:37.968745+0000","last_active":"2024-04-18T10:06:39.947596+0000","last_peered":"2024-04-18T10:06:39.947596+0000","last_clean":"2024-04-18T10:06:39.947596+0000","last_became_active":"2024-04-18T10:06:37.968364+0000","last_became_peered":"2024-04-18T10:06:37.968364+0000","last_unstale":"2024-04-18T10:06:39.947596+0000","last_undegraded":"2024-04-18T10:06:39.947596+0000","last_fullsized":"2024-04-18T10:06:39.947596+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:08:11.876955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.945555+0000","last_change":"2024-04-18T10:06:38.987092+0000","last_active":"2024-04-18T10:06:39.945555+0000","last_peered":"2024-04-18T10:06:39.945555+0000","last_clean":"2024-04-18T10:06:39.945555+0000","last_became_active":"2024-04-18T10:06:38.986004+0000","last_became_peered":"2024-04-18T10:06:38.986004+0000","last_unstale":"2024-04-18T10:06:39.945555+0000","last_undegraded":"2024-04-18T10:06:39.945555+0000","last_fullsized":"2024-04-18T10:06:39.945555+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:51:17.460536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947560+0000","last_change":"2024-04-18T10:06:37.961560+0000","last_active":"2024-04-18T10:06:39.947560+0000","last_peered":"2024-04-18T10:06:39.947560+0000","last_clean":"2024-04-18T10:06:39.947560+0000","last_became_active":"2024-04-18T10:06:37.960843+0000","last_became_peered":"2024-04-18T10:06:37.960843+0000","last_unstale":"2024-04-18T10:06:39.947560+0000","last_undegraded":"2024-04-18T10:06:39.947560+0000","last_fullsized":"2024-04-18T10:06:39.947560+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:25:40.656998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.946839+0000","last_change":"2024-04-18T10:06:38.978787+0000","last_active":"2024-04-18T10:06:39.946839+0000","last_peered":"2024-04-18T10:06:39.946839+0000","last_clean":"2024-04-18T10:06:39.946839+0000","last_became_active":"2024-04-18T10:06:38.977835+0000","last_became_peered":"2024-04-18T10:06:38.977835+0000","last_unstale":"2024-04-18T10:06:39.946839+0000","last_undegraded":"2024-04-18T10:06:39.946839+0000","last_fullsized":"2024-04-18T10:06:39.946839+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:44:02.108887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185332+0000","last_change":"2024-04-18T10:06:37.964766+0000","last_active":"2024-04-18T10:06:40.185332+0000","last_peered":"2024-04-18T10:06:40.185332+0000","last_clean":"2024-04-18T10:06:40.185332+0000","last_became_active":"2024-04-18T10:06:37.964444+0000","last_became_peered":"2024-04-18T10:06:37.964444+0000","last_unstale":"2024-04-18T10:06:40.185332+0000","last_undegraded":"2024-04-18T10:06:40.185332+0000","last_fullsized":"2024-04-18T10:06:40.185332+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:53:53.898473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947811+0000","last_change":"2024-04-18T10:06:38.996684+0000","last_active":"2024-04-18T10:06:39.947811+0000","last_peered":"2024-04-18T10:06:39.947811+0000","last_clean":"2024-04-18T10:06:39.947811+0000","last_became_active":"2024-04-18T10:06:38.996355+0000","last_became_peered":"2024-04-18T10:06:38.996355+0000","last_unstale":"2024-04-18T10:06:39.947811+0000","last_undegraded":"2024-04-18T10:06:39.947811+0000","last_fullsized":"2024-04-18T10:06:39.947811+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:52:40.978986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947121+0000","last_change":"2024-04-18T10:06:37.969359+0000","last_active":"2024-04-18T10:06:39.947121+0000","last_peered":"2024-04-18T10:06:39.947121+0000","last_clean":"2024-04-18T10:06:39.947121+0000","last_became_active":"2024-04-18T10:06:37.968586+0000","last_became_peered":"2024-04-18T10:06:37.968586+0000","last_unstale":"2024-04-18T10:06:39.947121+0000","last_undegraded":"2024-04-18T10:06:39.947121+0000","last_fullsized":"2024-04-18T10:06:39.947121+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:17:20.550946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.946570+0000","last_change":"2024-04-18T10:06:38.976030+0000","last_active":"2024-04-18T10:06:39.946570+0000","last_peered":"2024-04-18T10:06:39.946570+0000","last_clean":"2024-04-18T10:06:39.946570+0000","last_became_active":"2024-04-18T10:06:38.975298+0000","last_became_peered":"2024-04-18T10:06:38.975298+0000","last_unstale":"2024-04-18T10:06:39.946570+0000","last_undegraded":"2024-04-18T10:06:39.946570+0000","last_fullsized":"2024-04-18T10:06:39.946570+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:31:02.940425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186409+0000","last_change":"2024-04-18T10:06:37.974976+0000","last_active":"2024-04-18T10:06:40.186409+0000","last_peered":"2024-04-18T10:06:40.186409+0000","last_clean":"2024-04-18T10:06:40.186409+0000","last_became_active":"2024-04-18T10:06:37.970680+0000","last_became_peered":"2024-04-18T10:06:37.970680+0000","last_unstale":"2024-04-18T10:06:40.186409+0000","last_undegraded":"2024-04-18T10:06:40.186409+0000","last_fullsized":"2024-04-18T10:06:40.186409+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:18:22.685746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186357+0000","last_change":"2024-04-18T10:06:38.975138+0000","last_active":"2024-04-18T10:06:40.186357+0000","last_peered":"2024-04-18T10:06:40.186357+0000","last_clean":"2024-04-18T10:06:40.186357+0000","last_became_active":"2024-04-18T10:06:38.974708+0000","last_became_peered":"2024-04-18T10:06:38.974708+0000","last_unstale":"2024-04-18T10:06:40.186357+0000","last_undegraded":"2024-04-18T10:06:40.186357+0000","last_fullsized":"2024-04-18T10:06:40.186357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:47:38.632426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.944825+0000","last_change":"2024-04-18T10:06:37.963729+0000","last_active":"2024-04-18T10:06:39.944825+0000","last_peered":"2024-04-18T10:06:39.944825+0000","last_clean":"2024-04-18T10:06:39.944825+0000","last_became_active":"2024-04-18T10:06:37.962807+0000","last_became_peered":"2024-04-18T10:06:37.962807+0000","last_unstale":"2024-04-18T10:06:39.944825+0000","last_undegraded":"2024-04-18T10:06:39.944825+0000","last_fullsized":"2024-04-18T10:06:39.944825+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:24:02.490458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186057+0000","last_change":"2024-04-18T10:06:38.989841+0000","last_active":"2024-04-18T10:06:40.186057+0000","last_peered":"2024-04-18T10:06:40.186057+0000","last_clean":"2024-04-18T10:06:40.186057+0000","last_became_active":"2024-04-18T10:06:38.989606+0000","last_became_peered":"2024-04-18T10:06:38.989606+0000","last_unstale":"2024-04-18T10:06:40.186057+0000","last_undegraded":"2024-04-18T10:06:40.186057+0000","last_fullsized":"2024-04-18T10:06:40.186057+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:55:16.901514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"29'3","reported_seq":17,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948000+0000","last_change":"2024-04-18T10:06:37.971773+0000","last_active":"2024-04-18T10:06:39.948000+0000","last_peered":"2024-04-18T10:06:39.948000+0000","last_clean":"2024-04-18T10:06:39.948000+0000","last_became_active":"2024-04-18T10:06:37.971098+0000","last_became_peered":"2024-04-18T10:06:37.971098+0000","last_unstale":"2024-04-18T10:06:39.948000+0000","last_undegraded":"2024-04-18T10:06:39.948000+0000","last_fullsized":"2024-04-18T10:06:39.948000+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:32.616879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947334+0000","last_change":"2024-04-18T10:06:38.991217+0000","last_active":"2024-04-18T10:06:39.947334+0000","last_peered":"2024-04-18T10:06:39.947334+0000","last_clean":"2024-04-18T10:06:39.947334+0000","last_became_active":"2024-04-18T10:06:38.985951+0000","last_became_peered":"2024-04-18T10:06:38.985951+0000","last_unstale":"2024-04-18T10:06:39.947334+0000","last_undegraded":"2024-04-18T10:06:39.947334+0000","last_fullsized":"2024-04-18T10:06:39.947334+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:02:04.700746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"29'6","reported_seq":20,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186056+0000","last_change":"2024-04-18T10:06:37.969003+0000","last_active":"2024-04-18T10:06:40.186056+0000","last_peered":"2024-04-18T10:06:40.186056+0000","last_clean":"2024-04-18T10:06:40.186056+0000","last_became_active":"2024-04-18T10:06:37.966100+0000","last_became_peered":"2024-04-18T10:06:37.966100+0000","last_unstale":"2024-04-18T10:06:40.186056+0000","last_undegraded":"2024-04-18T10:06:40.186056+0000","last_fullsized":"2024-04-18T10:06:40.186056+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:39:36.237843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948936+0000","last_change":"2024-04-18T10:06:38.996726+0000","last_active":"2024-04-18T10:06:39.948936+0000","last_peered":"2024-04-18T10:06:39.948936+0000","last_clean":"2024-04-18T10:06:39.948936+0000","last_became_active":"2024-04-18T10:06:38.996443+0000","last_became_peered":"2024-04-18T10:06:38.996443+0000","last_unstale":"2024-04-18T10:06:39.948936+0000","last_undegraded":"2024-04-18T10:06:39.948936+0000","last_fullsized":"2024-04-18T10:06:39.948936+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:59:48.743845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947749+0000","last_change":"2024-04-18T10:06:37.970078+0000","last_active":"2024-04-18T10:06:39.947749+0000","last_peered":"2024-04-18T10:06:39.947749+0000","last_clean":"2024-04-18T10:06:39.947749+0000","last_became_active":"2024-04-18T10:06:37.969733+0000","last_became_peered":"2024-04-18T10:06:37.969733+0000","last_unstale":"2024-04-18T10:06:39.947749+0000","last_undegraded":"2024-04-18T10:06:39.947749+0000","last_fullsized":"2024-04-18T10:06:39.947749+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:22:31.962165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185879+0000","last_change":"2024-04-18T10:06:38.965884+0000","last_active":"2024-04-18T10:06:40.185879+0000","last_peered":"2024-04-18T10:06:40.185879+0000","last_clean":"2024-04-18T10:06:40.185879+0000","last_became_active":"2024-04-18T10:06:38.965508+0000","last_became_peered":"2024-04-18T10:06:38.965508+0000","last_unstale":"2024-04-18T10:06:40.185879+0000","last_undegraded":"2024-04-18T10:06:40.185879+0000","last_fullsized":"2024-04-18T10:06:40.185879+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:31:02.940425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948374+0000","last_change":"2024-04-18T10:06:37.972390+0000","last_active":"2024-04-18T10:06:39.948374+0000","last_peered":"2024-04-18T10:06:39.948374+0000","last_clean":"2024-04-18T10:06:39.948374+0000","last_became_active":"2024-04-18T10:06:37.972084+0000","last_became_peered":"2024-04-18T10:06:37.972084+0000","last_unstale":"2024-04-18T10:06:39.948374+0000","last_undegraded":"2024-04-18T10:06:39.948374+0000","last_fullsized":"2024-04-18T10:06:39.948374+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:47:47.427543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185390+0000","last_change":"2024-04-18T10:06:38.993859+0000","last_active":"2024-04-18T10:06:40.185390+0000","last_peered":"2024-04-18T10:06:40.185390+0000","last_clean":"2024-04-18T10:06:40.185390+0000","last_became_active":"2024-04-18T10:06:38.992501+0000","last_became_peered":"2024-04-18T10:06:38.992501+0000","last_unstale":"2024-04-18T10:06:40.185390+0000","last_undegraded":"2024-04-18T10:06:40.185390+0000","last_fullsized":"2024-04-18T10:06:40.185390+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:36:59.162611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185334+0000","last_change":"2024-04-18T10:06:37.970960+0000","last_active":"2024-04-18T10:06:40.185334+0000","last_peered":"2024-04-18T10:06:40.185334+0000","last_clean":"2024-04-18T10:06:40.185334+0000","last_became_active":"2024-04-18T10:06:37.970579+0000","last_became_peered":"2024-04-18T10:06:37.970579+0000","last_unstale":"2024-04-18T10:06:40.185334+0000","last_undegraded":"2024-04-18T10:06:40.185334+0000","last_fullsized":"2024-04-18T10:06:40.185334+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:17:20.550946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185780+0000","last_change":"2024-04-18T10:06:38.973388+0000","last_active":"2024-04-18T10:06:40.185780+0000","last_peered":"2024-04-18T10:06:40.185780+0000","last_clean":"2024-04-18T10:06:40.185780+0000","last_became_active":"2024-04-18T10:06:38.969323+0000","last_became_peered":"2024-04-18T10:06:38.969323+0000","last_unstale":"2024-04-18T10:06:40.185780+0000","last_undegraded":"2024-04-18T10:06:40.185780+0000","last_fullsized":"2024-04-18T10:06:40.185780+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:33.627030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"29'8","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.184142+0000","last_change":"2024-04-18T10:06:37.966039+0000","last_active":"2024-04-18T10:06:40.184142+0000","last_peered":"2024-04-18T10:06:40.184142+0000","last_clean":"2024-04-18T10:06:40.184142+0000","last_became_active":"2024-04-18T10:06:37.965749+0000","last_became_peered":"2024-04-18T10:06:37.965749+0000","last_unstale":"2024-04-18T10:06:40.184142+0000","last_undegraded":"2024-04-18T10:06:40.184142+0000","last_fullsized":"2024-04-18T10:06:40.184142+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:37:30.473092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948582+0000","last_change":"2024-04-18T10:06:38.993039+0000","last_active":"2024-04-18T10:06:39.948582+0000","last_peered":"2024-04-18T10:06:39.948582+0000","last_clean":"2024-04-18T10:06:39.948582+0000","last_became_active":"2024-04-18T10:06:38.992104+0000","last_became_peered":"2024-04-18T10:06:38.992104+0000","last_unstale":"2024-04-18T10:06:39.948582+0000","last_undegraded":"2024-04-18T10:06:39.948582+0000","last_fullsized":"2024-04-18T10:06:39.948582+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:51:17.460536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.949019+0000","last_change":"2024-04-18T10:06:37.962509+0000","last_active":"2024-04-18T10:06:39.949019+0000","last_peered":"2024-04-18T10:06:39.949019+0000","last_clean":"2024-04-18T10:06:39.949019+0000","last_became_active":"2024-04-18T10:06:37.962098+0000","last_became_peered":"2024-04-18T10:06:37.962098+0000","last_unstale":"2024-04-18T10:06:39.949019+0000","last_undegraded":"2024-04-18T10:06:39.949019+0000","last_fullsized":"2024-04-18T10:06:39.949019+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:08:11.876955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.946943+0000","last_change":"2024-04-18T10:06:38.993304+0000","last_active":"2024-04-18T10:06:39.946943+0000","last_peered":"2024-04-18T10:06:39.946943+0000","last_clean":"2024-04-18T10:06:39.946943+0000","last_became_active":"2024-04-18T10:06:38.992647+0000","last_became_peered":"2024-04-18T10:06:38.992647+0000","last_unstale":"2024-04-18T10:06:39.946943+0000","last_undegraded":"2024-04-18T10:06:39.946943+0000","last_fullsized":"2024-04-18T10:06:39.946943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:53:56.112173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948575+0000","last_change":"2024-04-18T10:06:37.970632+0000","last_active":"2024-04-18T10:06:39.948575+0000","last_peered":"2024-04-18T10:06:39.948575+0000","last_clean":"2024-04-18T10:06:39.948575+0000","last_became_active":"2024-04-18T10:06:37.969873+0000","last_became_peered":"2024-04-18T10:06:37.969873+0000","last_unstale":"2024-04-18T10:06:39.948575+0000","last_undegraded":"2024-04-18T10:06:39.948575+0000","last_fullsized":"2024-04-18T10:06:39.948575+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:39:36.237843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948322+0000","last_change":"2024-04-18T10:06:38.971522+0000","last_active":"2024-04-18T10:06:39.948322+0000","last_peered":"2024-04-18T10:06:39.948322+0000","last_clean":"2024-04-18T10:06:39.948322+0000","last_became_active":"2024-04-18T10:06:38.970919+0000","last_became_peered":"2024-04-18T10:06:38.970919+0000","last_unstale":"2024-04-18T10:06:39.948322+0000","last_undegraded":"2024-04-18T10:06:39.948322+0000","last_fullsized":"2024-04-18T10:06:39.948322+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:37:50.840199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"29'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948388+0000","last_change":"2024-04-18T10:06:37.964568+0000","last_active":"2024-04-18T10:06:39.948388+0000","last_peered":"2024-04-18T10:06:39.948388+0000","last_clean":"2024-04-18T10:06:39.948388+0000","last_became_active":"2024-04-18T10:06:37.963843+0000","last_became_peered":"2024-04-18T10:06:37.963843+0000","last_unstale":"2024-04-18T10:06:39.948388+0000","last_undegraded":"2024-04-18T10:06:39.948388+0000","last_fullsized":"2024-04-18T10:06:39.948388+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:13:24.715004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.184922+0000","last_change":"2024-04-18T10:06:38.998746+0000","last_active":"2024-04-18T10:06:40.184922+0000","last_peered":"2024-04-18T10:06:40.184922+0000","last_clean":"2024-04-18T10:06:40.184922+0000","last_became_active":"2024-04-18T10:06:38.998392+0000","last_became_peered":"2024-04-18T10:06:38.998392+0000","last_unstale":"2024-04-18T10:06:40.184922+0000","last_undegraded":"2024-04-18T10:06:40.184922+0000","last_fullsized":"2024-04-18T10:06:40.184922+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T19:39:37.247994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947935+0000","last_change":"2024-04-18T10:06:37.963538+0000","last_active":"2024-04-18T10:06:39.947935+0000","last_peered":"2024-04-18T10:06:39.947935+0000","last_clean":"2024-04-18T10:06:39.947935+0000","last_became_active":"2024-04-18T10:06:37.962801+0000","last_became_peered":"2024-04-18T10:06:37.962801+0000","last_unstale":"2024-04-18T10:06:39.947935+0000","last_undegraded":"2024-04-18T10:06:39.947935+0000","last_fullsized":"2024-04-18T10:06:39.947935+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:47:47.427543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.183603+0000","last_change":"2024-04-18T10:06:38.958713+0000","last_active":"2024-04-18T10:06:40.183603+0000","last_peered":"2024-04-18T10:06:40.183603+0000","last_clean":"2024-04-18T10:06:40.183603+0000","last_became_active":"2024-04-18T10:06:38.958378+0000","last_became_peered":"2024-04-18T10:06:38.958378+0000","last_unstale":"2024-04-18T10:06:40.183603+0000","last_undegraded":"2024-04-18T10:06:40.183603+0000","last_fullsized":"2024-04-18T10:06:40.183603+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:47:48.437694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.944345+0000","last_change":"2024-04-18T10:06:37.953110+0000","last_active":"2024-04-18T10:06:39.944345+0000","last_peered":"2024-04-18T10:06:39.944345+0000","last_clean":"2024-04-18T10:06:39.944345+0000","last_became_active":"2024-04-18T10:06:37.952833+0000","last_became_peered":"2024-04-18T10:06:37.952833+0000","last_unstale":"2024-04-18T10:06:39.944345+0000","last_undegraded":"2024-04-18T10:06:39.944345+0000","last_fullsized":"2024-04-18T10:06:39.944345+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:25:40.656998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.183888+0000","last_change":"2024-04-18T10:06:38.966474+0000","last_active":"2024-04-18T10:06:40.183888+0000","last_peered":"2024-04-18T10:06:40.183888+0000","last_clean":"2024-04-18T10:06:40.183888+0000","last_became_active":"2024-04-18T10:06:38.966074+0000","last_became_peered":"2024-04-18T10:06:38.966074+0000","last_unstale":"2024-04-18T10:06:40.183888+0000","last_undegraded":"2024-04-18T10:06:40.183888+0000","last_fullsized":"2024-04-18T10:06:40.183888+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:25:41.667149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185399+0000","last_change":"2024-04-18T10:06:37.975234+0000","last_active":"2024-04-18T10:06:40.185399+0000","last_peered":"2024-04-18T10:06:40.185399+0000","last_clean":"2024-04-18T10:06:40.185399+0000","last_became_active":"2024-04-18T10:06:37.974560+0000","last_became_peered":"2024-04-18T10:06:37.974560+0000","last_unstale":"2024-04-18T10:06:40.185399+0000","last_undegraded":"2024-04-18T10:06:40.185399+0000","last_fullsized":"2024-04-18T10:06:40.185399+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:47:47.427543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.183628+0000","last_change":"2024-04-18T10:06:38.960319+0000","last_active":"2024-04-18T10:06:40.183628+0000","last_peered":"2024-04-18T10:06:40.183628+0000","last_clean":"2024-04-18T10:06:40.183628+0000","last_became_active":"2024-04-18T10:06:38.959373+0000","last_became_peered":"2024-04-18T10:06:38.959373+0000","last_unstale":"2024-04-18T10:06:40.183628+0000","last_undegraded":"2024-04-18T10:06:40.183628+0000","last_fullsized":"2024-04-18T10:06:40.183628+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:24:03.500609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947576+0000","last_change":"2024-04-18T10:06:37.965584+0000","last_active":"2024-04-18T10:06:39.947576+0000","last_peered":"2024-04-18T10:06:39.947576+0000","last_clean":"2024-04-18T10:06:39.947576+0000","last_became_active":"2024-04-18T10:06:37.964535+0000","last_became_peered":"2024-04-18T10:06:37.964535+0000","last_unstale":"2024-04-18T10:06:39.947576+0000","last_undegraded":"2024-04-18T10:06:39.947576+0000","last_fullsized":"2024-04-18T10:06:39.947576+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:34:47.949029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.184799+0000","last_change":"2024-04-18T10:06:39.001074+0000","last_active":"2024-04-18T10:06:40.184799+0000","last_peered":"2024-04-18T10:06:40.184799+0000","last_clean":"2024-04-18T10:06:40.184799+0000","last_became_active":"2024-04-18T10:06:39.000638+0000","last_became_peered":"2024-04-18T10:06:39.000638+0000","last_unstale":"2024-04-18T10:06:40.184799+0000","last_undegraded":"2024-04-18T10:06:40.184799+0000","last_fullsized":"2024-04-18T10:06:40.184799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:22:50.350533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.945333+0000","last_change":"2024-04-18T10:06:37.962169+0000","last_active":"2024-04-18T10:06:39.945333+0000","last_peered":"2024-04-18T10:06:39.945333+0000","last_clean":"2024-04-18T10:06:39.945333+0000","last_became_active":"2024-04-18T10:06:37.961440+0000","last_became_peered":"2024-04-18T10:06:37.961440+0000","last_unstale":"2024-04-18T10:06:39.945333+0000","last_undegraded":"2024-04-18T10:06:39.945333+0000","last_fullsized":"2024-04-18T10:06:39.945333+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:28:45.478509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.184537+0000","last_change":"2024-04-18T10:06:38.952828+0000","last_active":"2024-04-18T10:06:40.184537+0000","last_peered":"2024-04-18T10:06:40.184537+0000","last_clean":"2024-04-18T10:06:40.184537+0000","last_became_active":"2024-04-18T10:06:38.952520+0000","last_became_peered":"2024-04-18T10:06:38.952520+0000","last_unstale":"2024-04-18T10:06:40.184537+0000","last_undegraded":"2024-04-18T10:06:40.184537+0000","last_fullsized":"2024-04-18T10:06:40.184537+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:17:21.561097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"29'2","reported_seq":16,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.949220+0000","last_change":"2024-04-18T10:06:37.970422+0000","last_active":"2024-04-18T10:06:39.949220+0000","last_peered":"2024-04-18T10:06:39.949220+0000","last_clean":"2024-04-18T10:06:39.949220+0000","last_became_active":"2024-04-18T10:06:37.969858+0000","last_became_peered":"2024-04-18T10:06:37.969858+0000","last_unstale":"2024-04-18T10:06:39.949220+0000","last_undegraded":"2024-04-18T10:06:39.949220+0000","last_fullsized":"2024-04-18T10:06:39.949220+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:25:40.656998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947317+0000","last_change":"2024-04-18T10:06:38.994574+0000","last_active":"2024-04-18T10:06:39.947317+0000","last_peered":"2024-04-18T10:06:39.947317+0000","last_clean":"2024-04-18T10:06:39.947317+0000","last_became_active":"2024-04-18T10:06:38.991709+0000","last_became_peered":"2024-04-18T10:06:38.991709+0000","last_unstale":"2024-04-18T10:06:39.947317+0000","last_undegraded":"2024-04-18T10:06:39.947317+0000","last_fullsized":"2024-04-18T10:06:39.947317+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:47:38.632426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"30'1","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947048+0000","last_change":"2024-04-18T10:06:37.954639+0000","last_active":"2024-04-18T10:06:39.947048+0000","last_peered":"2024-04-18T10:06:39.947048+0000","last_clean":"2024-04-18T10:06:39.947048+0000","last_became_active":"2024-04-18T10:06:37.953871+0000","last_became_peered":"2024-04-18T10:06:37.953871+0000","last_unstale":"2024-04-18T10:06:39.947048+0000","last_undegraded":"2024-04-18T10:06:39.947048+0000","last_fullsized":"2024-04-18T10:06:39.947048+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:08:11.876955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.943551+0000","last_change":"2024-04-18T10:06:38.974720+0000","last_active":"2024-04-18T10:06:39.943551+0000","last_peered":"2024-04-18T10:06:39.943551+0000","last_clean":"2024-04-18T10:06:39.943551+0000","last_became_active":"2024-04-18T10:06:38.974442+0000","last_became_peered":"2024-04-18T10:06:38.974442+0000","last_unstale":"2024-04-18T10:06:39.943551+0000","last_undegraded":"2024-04-18T10:06:39.943551+0000","last_fullsized":"2024-04-18T10:06:39.943551+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:37:31.483243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948587+0000","last_change":"2024-04-18T10:06:37.965931+0000","last_active":"2024-04-18T10:06:39.948587+0000","last_peered":"2024-04-18T10:06:39.948587+0000","last_clean":"2024-04-18T10:06:39.948587+0000","last_became_active":"2024-04-18T10:06:37.965099+0000","last_became_peered":"2024-04-18T10:06:37.965099+0000","last_unstale":"2024-04-18T10:06:39.948587+0000","last_undegraded":"2024-04-18T10:06:39.948587+0000","last_fullsized":"2024-04-18T10:06:39.948587+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:22:31.962165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185261+0000","last_change":"2024-04-18T10:06:38.962731+0000","last_active":"2024-04-18T10:06:40.185261+0000","last_peered":"2024-04-18T10:06:40.185261+0000","last_clean":"2024-04-18T10:06:40.185261+0000","last_became_active":"2024-04-18T10:06:38.962007+0000","last_became_peered":"2024-04-18T10:06:38.962007+0000","last_unstale":"2024-04-18T10:06:40.185261+0000","last_undegraded":"2024-04-18T10:06:40.185261+0000","last_fullsized":"2024-04-18T10:06:40.185261+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T17:53:54.908624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.944711+0000","last_change":"2024-04-18T10:06:37.958827+0000","last_active":"2024-04-18T10:06:39.944711+0000","last_peered":"2024-04-18T10:06:39.944711+0000","last_clean":"2024-04-18T10:06:39.944711+0000","last_became_active":"2024-04-18T10:06:37.958471+0000","last_became_peered":"2024-04-18T10:06:37.958471+0000","last_unstale":"2024-04-18T10:06:39.944711+0000","last_undegraded":"2024-04-18T10:06:39.944711+0000","last_fullsized":"2024-04-18T10:06:39.944711+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:22:31.962165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.944402+0000","last_change":"2024-04-18T10:06:38.969221+0000","last_active":"2024-04-18T10:06:39.944402+0000","last_peered":"2024-04-18T10:06:39.944402+0000","last_clean":"2024-04-18T10:06:39.944402+0000","last_became_active":"2024-04-18T10:06:38.968566+0000","last_became_peered":"2024-04-18T10:06:38.968566+0000","last_unstale":"2024-04-18T10:06:39.944402+0000","last_undegraded":"2024-04-18T10:06:39.944402+0000","last_fullsized":"2024-04-18T10:06:39.944402+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T14:52:40.978986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.948655+0000","last_change":"2024-04-18T10:06:37.960555+0000","last_active":"2024-04-18T10:06:39.948655+0000","last_peered":"2024-04-18T10:06:39.948655+0000","last_clean":"2024-04-18T10:06:39.948655+0000","last_became_active":"2024-04-18T10:06:37.958837+0000","last_became_peered":"2024-04-18T10:06:37.958837+0000","last_unstale":"2024-04-18T10:06:39.948655+0000","last_undegraded":"2024-04-18T10:06:39.948655+0000","last_fullsized":"2024-04-18T10:06:39.948655+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:24:02.490458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.183995+0000","last_change":"2024-04-18T10:06:38.958500+0000","last_active":"2024-04-18T10:06:40.183995+0000","last_peered":"2024-04-18T10:06:40.183995+0000","last_clean":"2024-04-18T10:06:40.183995+0000","last_became_active":"2024-04-18T10:06:38.957902+0000","last_became_peered":"2024-04-18T10:06:38.957902+0000","last_unstale":"2024-04-18T10:06:40.183995+0000","last_undegraded":"2024-04-18T10:06:40.183995+0000","last_fullsized":"2024-04-18T10:06:40.183995+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T10:51:17.460536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947823+0000","last_change":"2024-04-18T10:06:37.973900+0000","last_active":"2024-04-18T10:06:39.947823+0000","last_peered":"2024-04-18T10:06:39.947823+0000","last_clean":"2024-04-18T10:06:39.947823+0000","last_became_active":"2024-04-18T10:06:37.973066+0000","last_became_peered":"2024-04-18T10:06:37.973066+0000","last_unstale":"2024-04-18T10:06:39.947823+0000","last_undegraded":"2024-04-18T10:06:39.947823+0000","last_fullsized":"2024-04-18T10:06:39.947823+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T16:24:02.490458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:39.947278+0000","last_change":"2024-04-18T10:06:38.967825+0000","last_active":"2024-04-18T10:06:39.947278+0000","last_peered":"2024-04-18T10:06:39.947278+0000","last_clean":"2024-04-18T10:06:39.947278+0000","last_became_active":"2024-04-18T10:06:38.967071+0000","last_became_peered":"2024-04-18T10:06:38.967071+0000","last_unstale":"2024-04-18T10:06:39.947278+0000","last_undegraded":"2024-04-18T10:06:39.947278+0000","last_fullsized":"2024-04-18T10:06:39.947278+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T20:59:48.743845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.184267+0000","last_change":"2024-04-18T10:06:37.970981+0000","last_active":"2024-04-18T10:06:40.184267+0000","last_peered":"2024-04-18T10:06:40.184267+0000","last_clean":"2024-04-18T10:06:40.184267+0000","last_became_active":"2024-04-18T10:06:37.970492+0000","last_became_peered":"2024-04-18T10:06:37.970492+0000","last_unstale":"2024-04-18T10:06:40.184267+0000","last_undegraded":"2024-04-18T10:06:40.184267+0000","last_fullsized":"2024-04-18T10:06:40.184267+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T12:57:32.616879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.186536+0000","last_change":"2024-04-18T10:06:38.998473+0000","last_active":"2024-04-18T10:06:40.186536+0000","last_peered":"2024-04-18T10:06:40.186536+0000","last_clean":"2024-04-18T10:06:40.186536+0000","last_became_active":"2024-04-18T10:06:38.990544+0000","last_became_peered":"2024-04-18T10:06:38.990544+0000","last_unstale":"2024-04-18T10:06:40.186536+0000","last_undegraded":"2024-04-18T10:06:40.186536+0000","last_fullsized":"2024-04-18T10:06:40.186536+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:37.925903+0000","last_clean_scrub_stamp":"2024-04-18T10:06:37.925903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T21:55:16.901514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.184978+0000","last_change":"2024-04-18T10:06:37.975972+0000","last_active":"2024-04-18T10:06:40.184978+0000","last_peered":"2024-04-18T10:06:40.184978+0000","last_clean":"2024-04-18T10:06:40.184978+0000","last_became_active":"2024-04-18T10:06:37.975612+0000","last_became_peered":"2024-04-18T10:06:37.975612+0000","last_unstale":"2024-04-18T10:06:40.184978+0000","last_undegraded":"2024-04-18T10:06:40.184978+0000","last_fullsized":"2024-04-18T10:06:40.184978+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":27,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:06:36.915753+0000","last_clean_scrub_stamp":"2024-04-18T10:06:36.915753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T15:34:47.949029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":186,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-04-18T10:06:40.185353+0000","last_change":"2024-04-18T10:05:49.183850+0000","last_active":"2024-04-18T10:06:40.185353+0000","last_peered":"2024-04-18T10:06:40.185353+0000","last_clean":"2024-04-18T10:06:40.185353+0000","last_became_active":"2024-04-18T10:05:49.181471+0000","last_became_peered":"2024-04-18T10:05:49.181471+0000","last_unstale":"2024-04-18T10:06:40.185353+0000","last_undegraded":"2024-04-18T10:06:40.185353+0000","last_fullsized":"2024-04-18T10:06:40.185353+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_clean_scrub_stamp":"2024-04-18T10:05:47.511486+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:06:51.214547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":933888,"data_stored":719141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":344064,"data_stored":129299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":344064,"data_stored":129355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574864,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":344064,"data_stored":128787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":352256,"data_stored":130317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":339968,"data_stored":128753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":352256,"data_stored":130317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574862,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27932,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343908,"statfs":{"total":96636764160,"available":96608161792,"internally_reserved":0,"allocated":929792,"data_stored":719051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T10:06:43.375 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-18T10:06:43.375 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-18T10:06:43.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.541+0000 7f5c998fd640 1 Processor -- start 2024-04-18T10:06:43.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.541+0000 7f5c998fd640 1 -- start start 2024-04-18T10:06:43.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.541+0000 7f5c998fd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5c940b5270 0x7f5c9413c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.541+0000 7f5c998fd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5c9413c780 0x7f5c94145d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.541+0000 7f5c998fd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c940b29d0 0x7f5c94146270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.541+0000 7f5c998fd640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f5c94074060 con 0x7f5c940b5270 2024-04-18T10:06:43.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.541+0000 7f5c998fd640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f5c940b4590 con 0x7f5c9413c780 2024-04-18T10:06:43.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.541+0000 7f5c998fd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f5c940b4870 con 0x7f5c940b29d0 2024-04-18T10:06:43.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c92ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5c940b5270 0x7f5c9413c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c92ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f5c940b5270 0x7f5c9413c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56438/0 (socket says 172.21.15.62:56438) 2024-04-18T10:06:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c92ffd640 1 -- 172.21.15.62:0/3622413994 learned_addr learned my addr 172.21.15.62:0/3622413994 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c927fc640 1 --2- 172.21.15.62:0/3622413994 >> v2:172.21.15.145:3300/0 conn(0x7f5c9413c780 0x7f5c94145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c937fe640 1 --2- 172.21.15.62:0/3622413994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c940b29d0 0x7f5c94146270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c92ffd640 1 -- 172.21.15.62:0/3622413994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c940b29d0 msgr2=0x7f5c94146270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c92ffd640 1 --2- 172.21.15.62:0/3622413994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c940b29d0 0x7f5c94146270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c92ffd640 1 -- 172.21.15.62:0/3622413994 >> v2:172.21.15.145:3300/0 conn(0x7f5c9413c780 msgr2=0x7f5c94145d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c92ffd640 1 --2- 172.21.15.62:0/3622413994 >> v2:172.21.15.145:3300/0 conn(0x7f5c9413c780 0x7f5c94145d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c92ffd640 1 -- 172.21.15.62:0/3622413994 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c940b4b50 con 0x7f5c940b5270 2024-04-18T10:06:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c927fc640 1 --2- 172.21.15.62:0/3622413994 >> v2:172.21.15.145:3300/0 conn(0x7f5c9413c780 0x7f5c94145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:43.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c937fe640 1 --2- 172.21.15.62:0/3622413994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c940b29d0 0x7f5c94146270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:43.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c92ffd640 1 --2- 172.21.15.62:0/3622413994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c940b5270 0x7f5c9413c240 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5c94145a50 tx=0x7f5c8401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed427f481c5e6851 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:43.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c91ffb640 1 -- 172.21.15.62:0/3622413994 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c840a7070 con 0x7f5c940b5270 2024-04-18T10:06:43.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c91ffb640 1 -- 172.21.15.62:0/3622413994 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c840b0070 con 0x7f5c940b5270 2024-04-18T10:06:43.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c91ffb640 1 -- 172.21.15.62:0/3622413994 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c940b4350 con 0x7f5c940b5270 2024-04-18T10:06:43.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c998fd640 1 -- 172.21.15.62:0/3622413994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c940b5270 msgr2=0x7f5c9413c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c998fd640 1 --2- 172.21.15.62:0/3622413994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c940b5270 0x7f5c9413c240 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f5c94145a50 tx=0x7f5c8401c490 comp rx=0 tx=0).stop 2024-04-18T10:06:43.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c998fd640 1 -- 172.21.15.62:0/3622413994 shutdown_connections 2024-04-18T10:06:43.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c998fd640 1 --2- 172.21.15.62:0/3622413994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c940b29d0 0x7f5c94146270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c998fd640 1 --2- 172.21.15.62:0/3622413994 >> v2:172.21.15.145:3300/0 conn(0x7f5c9413c780 0x7f5c94145d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c998fd640 1 --2- 172.21.15.62:0/3622413994 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c940b5270 0x7f5c9413c240 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c998fd640 1 -- 172.21.15.62:0/3622413994 >> 172.21.15.62:0/3622413994 conn(0x7f5c940bb4d0 msgr2=0x7f5c940bb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:43.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c998fd640 1 -- 172.21.15.62:0/3622413994 shutdown_connections 2024-04-18T10:06:43.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.545+0000 7f5c998fd640 1 -- 172.21.15.62:0/3622413994 wait complete. 2024-04-18T10:06:43.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c998fd640 1 Processor -- start 2024-04-18T10:06:43.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c998fd640 1 -- start start 2024-04-18T10:06:43.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c998fd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c940b29d0 0x7f5c941de730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c998fd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c940b5270 0x7f5c940baa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c998fd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c941409b0 0x7f5c94140e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c998fd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94149be0 con 0x7f5c941409b0 2024-04-18T10:06:43.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c998fd640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94149a60 con 0x7f5c940b5270 2024-04-18T10:06:43.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c998fd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f5c94149d60 con 0x7f5c940b29d0 2024-04-18T10:06:43.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c937fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c941409b0 0x7f5c94140e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c937fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c941409b0 0x7f5c94140e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56446/0 (socket says 172.21.15.62:56446) 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c937fe640 1 -- 172.21.15.62:0/1655559377 learned_addr learned my addr 172.21.15.62:0/1655559377 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c92ffd640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c940b29d0 0x7f5c941de730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c927fc640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c940b5270 0x7f5c940baa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c937fe640 1 -- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c940b29d0 msgr2=0x7f5c941de730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c937fe640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c940b29d0 0x7f5c941de730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c937fe640 1 -- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c940b5270 msgr2=0x7f5c940baa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c937fe640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c940b5270 0x7f5c940baa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c937fe640 1 -- 172.21.15.62:0/1655559377 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c94142c60 con 0x7f5c941409b0 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c937fe640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c941409b0 0x7f5c94140e60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00cdf0 tx=0x7f5c8c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c7bfff640 1 -- 172.21.15.62:0/1655559377 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c003ba0 con 0x7f5c941409b0 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c7bfff640 1 -- 172.21.15.62:0/1655559377 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c8c003d60 con 0x7f5c941409b0 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c998fd640 1 -- 172.21.15.62:0/1655559377 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c941413a0 con 0x7f5c941409b0 2024-04-18T10:06:43.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c7bfff640 1 -- 172.21.15.62:0/1655559377 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c00d880 con 0x7f5c941409b0 2024-04-18T10:06:43.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c998fd640 1 -- 172.21.15.62:0/1655559377 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c941df030 con 0x7f5c941409b0 2024-04-18T10:06:43.552 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c998fd640 1 -- 172.21.15.62:0/1655559377 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c94069230 con 0x7f5c941409b0 2024-04-18T10:06:43.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.549+0000 7f5c7bfff640 1 -- 172.21.15.62:0/1655559377 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5c8c00da20 con 0x7f5c941409b0 2024-04-18T10:06:43.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.553+0000 7f5c7bfff640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5c80068380 0x7f5c8006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.553+0000 7f5c7bfff640 1 -- 172.21.15.62:0/1655559377 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f5c8c0879a0 con 0x7f5c941409b0 2024-04-18T10:06:43.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.553+0000 7f5c92ffd640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5c80068380 0x7f5c8006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.553 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.553+0000 7f5c92ffd640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5c80068380 0x7f5c8006a840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5c840af000 tx=0x7f5c840a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:43.555 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.557+0000 7f5c7bfff640 1 -- 172.21.15.62:0/1655559377 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5c8c0580d0 con 0x7f5c941409b0 2024-04-18T10:06:43.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.777+0000 7f5c998fd640 1 -- 172.21.15.62:0/1655559377 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f5c94144180 con 0x7f5c941409b0 2024-04-18T10:06:43.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c7bfff640 1 -- 172.21.15.62:0/1655559377 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f5c8c0579d0 con 0x7f5c941409b0 2024-04-18T10:06:43.780 INFO:teuthology.orchestra.run.smithi062.stdout:[client.0] 2024-04-18T10:06:43.780 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQBN8CBmSFtIDRAAjl9lXubksjAuIJZdtH8x2Q== 2024-04-18T10:06:43.780 INFO:teuthology.orchestra.run.smithi062.stdout: caps mds = "allow" 2024-04-18T10:06:43.780 INFO:teuthology.orchestra.run.smithi062.stdout: caps mon = "allow r" 2024-04-18T10:06:43.780 INFO:teuthology.orchestra.run.smithi062.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-18T10:06:43.781 INFO:teuthology.orchestra.run.smithi062.stderr:updated caps for client.0 2024-04-18T10:06:43.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c998fd640 1 -- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5c80068380 msgr2=0x7f5c8006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c998fd640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5c80068380 0x7f5c8006a840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5c840af000 tx=0x7f5c840a7000 comp rx=0 tx=0).stop 2024-04-18T10:06:43.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c998fd640 1 -- 172.21.15.62:0/1655559377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c941409b0 msgr2=0x7f5c94140e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c998fd640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c941409b0 0x7f5c94140e60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00cdf0 tx=0x7f5c8c00a710 comp rx=0 tx=0).stop 2024-04-18T10:06:43.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c998fd640 1 -- 172.21.15.62:0/1655559377 shutdown_connections 2024-04-18T10:06:43.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c998fd640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f5c80068380 0x7f5c8006a840 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c998fd640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c941409b0 0x7f5c94140e60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c998fd640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c940b5270 0x7f5c940baa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c998fd640 1 --2- 172.21.15.62:0/1655559377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c940b29d0 0x7f5c941de730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c998fd640 1 -- 172.21.15.62:0/1655559377 >> 172.21.15.62:0/1655559377 conn(0x7f5c940bb4d0 msgr2=0x7f5c94135470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:43.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c998fd640 1 -- 172.21.15.62:0/1655559377 shutdown_connections 2024-04-18T10:06:43.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.781+0000 7f5c998fd640 1 -- 172.21.15.62:0/1655559377 wait complete. 2024-04-18T10:06:43.800 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:43.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf79d43640 1 Processor -- start 2024-04-18T10:06:43.916 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf79d43640 1 -- start start 2024-04-18T10:06:43.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf79d43640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdf7409be50 0x7fdf74145720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf79d43640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf740b5550 con 0x7fdf740b9d90 2024-04-18T10:06:43.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf79d43640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7409e370 con 0x7fdf740b5100 2024-04-18T10:06:43.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf79d43640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fdf740b4190 con 0x7fdf7409be50 2024-04-18T10:06:43.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf737fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fdf740b9d90 0x7fdf74145010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42888/0 (socket says 172.21.15.62:42888) 2024-04-18T10:06:43.917 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf737fe640 1 -- 172.21.15.62:0/1007681343 learned_addr learned my addr 172.21.15.62:0/1007681343 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:43.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf72ffd640 1 --2- 172.21.15.62:0/1007681343 >> v2:172.21.15.145:3300/0 conn(0x7fdf7409be50 0x7fdf74145720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 283416703 0 0) 0x7fdf740b5550 con 0x7fdf740b9d90 2024-04-18T10:06:43.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf60003530 con 0x7fdf740b9d90 2024-04-18T10:06:43.919 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3984429596 0 0) 0x7fdf60003530 con 0x7fdf740b9d90 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 >> v2:172.21.15.145:3300/0 conn(0x7fdf7409be50 msgr2=0x7fdf74145720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 --2- 172.21.15.62:0/1007681343 >> v2:172.21.15.145:3300/0 conn(0x7fdf7409be50 0x7fdf74145720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 >> v1:172.21.15.145:6789/0 conn(0x7fdf740b5100 legacy=0x7fdf741504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf740ba170 con 0x7fdf740b9d90 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fdf64001680 con 0x7fdf740b9d90 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 >> v1:172.21.15.62:6789/0 conn(0x7fdf740b9d90 legacy=0x7fdf74145010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 --2- 172.21.15.62:0/1007681343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf600039c0 0x7fdf60003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 --2- 172.21.15.62:0/1007681343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 0x7fdf60004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 --2- 172.21.15.62:0/1007681343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf600051d0 0x7fdf6000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdf740b4190 con 0x7fdf60004810 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6000c170 con 0x7fdf600051d0 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fdf6000c370 con 0x7fdf600039c0 2024-04-18T10:06:43.920 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf72ffd640 1 --2- 172.21.15.62:0/1007681343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 0x7fdf60004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf737fe640 1 --2- 172.21.15.62:0/1007681343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf600039c0 0x7fdf60003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf73fff640 1 --2- 172.21.15.62:0/1007681343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf600051d0 0x7fdf6000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf72ffd640 1 -- 172.21.15.62:0/1007681343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf600039c0 msgr2=0x7fdf60003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:43.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf72ffd640 1 --2- 172.21.15.62:0/1007681343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf600039c0 0x7fdf60003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf72ffd640 1 -- 172.21.15.62:0/1007681343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf600051d0 msgr2=0x7fdf6000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:43.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf72ffd640 1 --2- 172.21.15.62:0/1007681343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf600051d0 0x7fdf6000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf72ffd640 1 -- 172.21.15.62:0/1007681343 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf60003530 con 0x7fdf60004810 2024-04-18T10:06:43.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf73fff640 1 --2- 172.21.15.62:0/1007681343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf600051d0 0x7fdf6000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:43.921 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.917+0000 7fdf73fff640 1 -- 172.21.15.62:0/1007681343 reap_dead start 2024-04-18T10:06:43.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf72ffd640 1 --2- 172.21.15.62:0/1007681343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 0x7fdf60004c90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009ab0 tx=0x7fdf5c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962f682eac06902f server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:43.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0096b0 con 0x7fdf60004810 2024-04-18T10:06:43.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf5c0a6480 con 0x7fdf60004810 2024-04-18T10:06:43.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf727fc640 1 -- 172.21.15.62:0/1007681343 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c0a6870 con 0x7fdf60004810 2024-04-18T10:06:43.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 -- 172.21.15.62:0/1007681343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 msgr2=0x7fdf60004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 --2- 172.21.15.62:0/1007681343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 0x7fdf60004c90 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009ab0 tx=0x7fdf5c01ca50 comp rx=0 tx=0).stop 2024-04-18T10:06:43.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 -- 172.21.15.62:0/1007681343 shutdown_connections 2024-04-18T10:06:43.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 --2- 172.21.15.62:0/1007681343 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 0x7fdf60004c90 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 -- 172.21.15.62:0/1007681343 >> 172.21.15.62:0/1007681343 conn(0x7fdf7406c0d0 msgr2=0x7fdf740b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:43.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 -- 172.21.15.62:0/1007681343 shutdown_connections 2024-04-18T10:06:43.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 -- 172.21.15.62:0/1007681343 wait complete. 2024-04-18T10:06:43.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 Processor -- start 2024-04-18T10:06:43.923 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 -- start start 2024-04-18T10:06:43.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 0x7fdf740afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf74098820 0x7fdf741de410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf74098ca0 0x7fdf741de950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fdf741459a0 con 0x7fdf60004810 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fdf74145820 con 0x7fdf74098820 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf79d43640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fdf7406d210 con 0x7fdf74098ca0 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf737fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 0x7fdf740afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf737fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 0x7fdf740afc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56474/0 (socket says 172.21.15.62:56474) 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf737fe640 1 -- 172.21.15.62:0/673895108 learned_addr learned my addr 172.21.15.62:0/673895108 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf72ffd640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf74098820 0x7fdf741de410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf73fff640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf74098ca0 0x7fdf741de950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf737fe640 1 -- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf74098ca0 msgr2=0x7fdf741de950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf737fe640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf74098ca0 0x7fdf741de950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf737fe640 1 -- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf74098820 msgr2=0x7fdf741de410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:43.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf737fe640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf74098820 0x7fdf741de410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:43.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.921+0000 7fdf737fe640 1 -- 172.21.15.62:0/673895108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf74070ab0 con 0x7fdf60004810 2024-04-18T10:06:43.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf72ffd640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf74098820 0x7fdf741de410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:43.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf737fe640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 0x7fdf740afc70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400e7b0 tx=0x7fdf6400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:43.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf6bfff640 1 -- 172.21.15.62:0/673895108 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6400c8d0 con 0x7fdf60004810 2024-04-18T10:06:43.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf6bfff640 1 -- 172.21.15.62:0/673895108 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf64018070 con 0x7fdf60004810 2024-04-18T10:06:43.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf79d43640 1 -- 172.21.15.62:0/673895108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7413e710 con 0x7fdf60004810 2024-04-18T10:06:43.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf6bfff640 1 -- 172.21.15.62:0/673895108 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6400ccd0 con 0x7fdf60004810 2024-04-18T10:06:43.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf79d43640 1 -- 172.21.15.62:0/673895108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf74149d60 con 0x7fdf60004810 2024-04-18T10:06:43.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf79d43640 1 -- 172.21.15.62:0/673895108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf74068b90 con 0x7fdf60004810 2024-04-18T10:06:43.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf6bfff640 1 -- 172.21.15.62:0/673895108 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdf6401d460 con 0x7fdf60004810 2024-04-18T10:06:43.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf6bfff640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdf54068380 0x7fdf5406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:43.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf6bfff640 1 -- 172.21.15.62:0/673895108 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fdf64086880 con 0x7fdf60004810 2024-04-18T10:06:43.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf72ffd640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdf54068380 0x7fdf5406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:43.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.925+0000 7fdf72ffd640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdf54068380 0x7fdf5406a840 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0ad000 tx=0x7fdf5c0046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:43.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:43.929+0000 7fdf6bfff640 1 -- 172.21.15.62:0/673895108 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdf64056ed0 con 0x7fdf60004810 2024-04-18T10:06:44.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 -- 172.21.15.62:0/673895108 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf74073540 con 0x7fdf60004810 2024-04-18T10:06:44.124 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf6bfff640 1 -- 172.21.15.62:0/673895108 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+3534 (secure 0 0 0) 0x7fdf640567d0 con 0x7fdf60004810 2024-04-18T10:06:44.125 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:44.125 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4705,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1649467687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1649467687},{"type":"v1","addr":"172.21.15.145:6835","nonce":1649467687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:06:38.304481+0000","modified":"2024-04-18T10:06:41.339980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4932":{"gid":4932,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2971073685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2971073685},{"type":"v1","addr":"172.21.15.62:6833","nonce":2971073685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4932,"qdb_cluster":[4932]},"id":2}]} 2024-04-18T10:06:44.125 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-04-18T10:06:44.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 -- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdf54068380 msgr2=0x7fdf5406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdf54068380 0x7fdf5406a840 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c0ad000 tx=0x7fdf5c0046d0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 -- 172.21.15.62:0/673895108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 msgr2=0x7fdf740afc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 0x7fdf740afc70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400e7b0 tx=0x7fdf6400ec70 comp rx=0 tx=0).stop 2024-04-18T10:06:44.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 -- 172.21.15.62:0/673895108 shutdown_connections 2024-04-18T10:06:44.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fdf54068380 0x7fdf5406a840 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf74098ca0 0x7fdf741de950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf74098820 0x7fdf741de410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 --2- 172.21.15.62:0/673895108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdf60004810 0x7fdf740afc70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 -- 172.21.15.62:0/673895108 >> 172.21.15.62:0/673895108 conn(0x7fdf7406c0d0 msgr2=0x7fdf74146540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:44.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 -- 172.21.15.62:0/673895108 shutdown_connections 2024-04-18T10:06:44.128 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.125+0000 7fdf79d43640 1 -- 172.21.15.62:0/673895108 wait complete. 2024-04-18T10:06:44.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-18T10:06:38.304481+0000', 'modified': '2024-04-18T10:06:41.339980+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4932}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4932': {'gid': 4932, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.62:6833/2971073685', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6832', 'nonce': 2971073685}, {'type': 'v1', 'addr': '172.21.15.62:6833', 'nonce': 2971073685}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4932, 'qdb_cluster': [4932]} 2024-04-18T10:06:44.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-18T10:06:44.348 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-18T10:06:44.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47eac0a640 1 Processor -- start 2024-04-18T10:06:44.477 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47eac0a640 1 -- start start 2024-04-18T10:06:44.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47eac0a640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f47e409b620 0x7f47e409ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47eac0a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e409d430 con 0x7f47e409e2d0 2024-04-18T10:06:44.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47eac0a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e409dea0 con 0x7f47e409c4d0 2024-04-18T10:06:44.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47eac0a640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f47e406eb30 con 0x7f47e409b620 2024-04-18T10:06:44.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47e89a6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f47e409b620 0x7f47e409ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47e89a6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f47e409b620 0x7f47e409ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56488/0 (socket says 172.21.15.62:56488) 2024-04-18T10:06:44.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47e89a6640 1 -- 172.21.15.62:0/3748382127 learned_addr learned my addr 172.21.15.62:0/3748382127 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:44.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47e91a7640 1 --1- 172.21.15.62:0/3748382127 >> v1:172.21.15.62:6789/0 conn(0x7f47e409e2d0 0x7f47e4142ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42890/0 (socket says 172.21.15.62:42890) 2024-04-18T10:06:44.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47e37fe640 1 -- 172.21.15.62:0/3748382127 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 256179562 0 0) 0x7f47e409d430 con 0x7f47e409e2d0 2024-04-18T10:06:44.479 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47e37fe640 1 -- 172.21.15.62:0/3748382127 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47cc003530 con 0x7f47e409e2d0 2024-04-18T10:06:44.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47e89a6640 1 -- 172.21.15.62:0/3748382127 >> v1:172.21.15.145:6789/0 conn(0x7f47e409c4d0 legacy=0x7f47e41492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:44.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47e89a6640 1 -- 172.21.15.62:0/3748382127 >> v1:172.21.15.62:6789/0 conn(0x7f47e409e2d0 legacy=0x7f47e4142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.480 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.477+0000 7f47e89a6640 1 -- 172.21.15.62:0/3748382127 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e409d6a0 con 0x7f47e409b620 2024-04-18T10:06:44.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47e89a6640 1 --2- 172.21.15.62:0/3748382127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47e409b620 0x7f47e409ba00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f47d40098a0 tx=0x7f47d401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e7d4bd80e34015 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:44.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47e37fe640 1 -- 172.21.15.62:0/3748382127 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d40b0070 con 0x7f47e409b620 2024-04-18T10:06:44.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47e37fe640 1 -- 172.21.15.62:0/3748382127 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d40ab600 con 0x7f47e409b620 2024-04-18T10:06:44.481 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47e37fe640 1 -- 172.21.15.62:0/3748382127 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d40ab9b0 con 0x7f47e409b620 2024-04-18T10:06:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 -- 172.21.15.62:0/3748382127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47e409b620 msgr2=0x7f47e409ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 --2- 172.21.15.62:0/3748382127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47e409b620 0x7f47e409ba00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f47d40098a0 tx=0x7f47d401c470 comp rx=0 tx=0).stop 2024-04-18T10:06:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 -- 172.21.15.62:0/3748382127 shutdown_connections 2024-04-18T10:06:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 --2- 172.21.15.62:0/3748382127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47e409b620 0x7f47e409ba00 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 -- 172.21.15.62:0/3748382127 >> 172.21.15.62:0/3748382127 conn(0x7f47e40b9420 msgr2=0x7f47e40b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 -- 172.21.15.62:0/3748382127 shutdown_connections 2024-04-18T10:06:44.482 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 -- 172.21.15.62:0/3748382127 wait complete. 2024-04-18T10:06:44.484 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 Processor -- start 2024-04-18T10:06:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 -- start start 2024-04-18T10:06:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e409b620 0x7f47e4071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47e409c4d0 0x7f47e413e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e409e2d0 0x7f47e41e1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f47e4153ba0 con 0x7f47e409c4d0 2024-04-18T10:06:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f47e4153a20 con 0x7f47e409b620 2024-04-18T10:06:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.481+0000 7f47eac0a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f47e4153d20 con 0x7f47e409e2d0 2024-04-18T10:06:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47e409c4d0 0x7f47e413e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47e409c4d0 0x7f47e413e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56502/0 (socket says 172.21.15.62:56502) 2024-04-18T10:06:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e3fff640 1 -- 172.21.15.62:0/1106218201 learned_addr learned my addr 172.21.15.62:0/1106218201 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e91a7640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e409e2d0 0x7f47e41e1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.485 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e89a6640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e409b620 0x7f47e4071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e3fff640 1 -- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e409e2d0 msgr2=0x7f47e41e1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e3fff640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e409e2d0 0x7f47e41e1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e3fff640 1 -- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e409b620 msgr2=0x7f47e4071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e3fff640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e409b620 0x7f47e4071990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e3fff640 1 -- 172.21.15.62:0/1106218201 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e4070ea0 con 0x7f47e409c4d0 2024-04-18T10:06:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e3fff640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47e409c4d0 0x7f47e413e420 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f47d800c430 tx=0x7f47d800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e17fa640 1 -- 172.21.15.62:0/1106218201 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d8007020 con 0x7f47e409c4d0 2024-04-18T10:06:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e17fa640 1 -- 172.21.15.62:0/1106218201 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d80071c0 con 0x7f47e409c4d0 2024-04-18T10:06:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47eac0a640 1 -- 172.21.15.62:0/1106218201 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e40706a0 con 0x7f47e409c4d0 2024-04-18T10:06:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e17fa640 1 -- 172.21.15.62:0/1106218201 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d80075b0 con 0x7f47e409c4d0 2024-04-18T10:06:44.486 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47eac0a640 1 -- 172.21.15.62:0/1106218201 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e40b2950 con 0x7f47e409c4d0 2024-04-18T10:06:44.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e17fa640 1 -- 172.21.15.62:0/1106218201 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f47d8019020 con 0x7f47e409c4d0 2024-04-18T10:06:44.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e17fa640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f47d00683b0 0x7f47d006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e17fa640 1 -- 172.21.15.62:0/1106218201 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f47d80860e0 con 0x7f47e409c4d0 2024-04-18T10:06:44.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47e89a6640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f47d00683b0 0x7f47d006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47edd8c000 1 Processor -- start 2024-04-18T10:06:44.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.485+0000 7f47edd8c000 1 -- start start 2024-04-18T10:06:44.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 0x56371e4fa060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4fa5a0 0x56371e5249d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e524f10 0x56371e5252f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56371e3db6f0 con 0x56371e4fa5a0 2024-04-18T10:06:44.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x56371e3424d0 con 0x56371e524f10 2024-04-18T10:06:44.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x56371e2dcbf0 con 0x56371e4f9c80 2024-04-18T10:06:44.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e89a6640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f47d00683b0 0x7f47d006a870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f47d40ad000 tx=0x7f47d40a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:44.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4fa5a0 0x56371e5249d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4fa5a0 0x56371e5249d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56514/0 (socket says 172.21.15.62:56514) 2024-04-18T10:06:44.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e3fff640 1 -- 172.21.15.62:0/310496289 learned_addr learned my addr 172.21.15.62:0/310496289 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:44.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e91a7640 1 --2- 172.21.15.62:0/310496289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e524f10 0x56371e5252f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e89a6640 1 --2- 172.21.15.62:0/310496289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 0x56371e4fa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e3fff640 1 -- 172.21.15.62:0/310496289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 msgr2=0x56371e4fa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:44.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e3fff640 1 --2- 172.21.15.62:0/310496289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 0x56371e4fa060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e3fff640 1 -- 172.21.15.62:0/310496289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e524f10 msgr2=0x56371e5252f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e3fff640 1 --2- 172.21.15.62:0/310496289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e524f10 0x56371e5252f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e3fff640 1 -- 172.21.15.62:0/310496289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56371e2dce40 con 0x56371e4fa5a0 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e89a6640 1 --2- 172.21.15.62:0/310496289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 0x56371e4fa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e3fff640 1 --2- 172.21.15.62:0/310496289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4fa5a0 0x56371e5249d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f47d800a040 tx=0x7f47d8055a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7832a908ab5fee1 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47ca7fc640 1 -- 172.21.15.62:0/310496289 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d8082070 con 0x56371e4fa5a0 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47ca7fc640 1 -- 172.21.15.62:0/310496289 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d805ec10 con 0x56371e4fa5a0 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47ca7fc640 1 -- 172.21.15.62:0/310496289 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d806e070 con 0x56371e4fa5a0 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- 172.21.15.62:0/310496289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4fa5a0 msgr2=0x56371e5249d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 --2- 172.21.15.62:0/310496289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4fa5a0 0x56371e5249d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f47d800a040 tx=0x7f47d8055a90 comp rx=0 tx=0).stop 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- 172.21.15.62:0/310496289 shutdown_connections 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 --2- 172.21.15.62:0/310496289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e524f10 0x56371e5252f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 --2- 172.21.15.62:0/310496289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4fa5a0 0x56371e5249d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 --2- 172.21.15.62:0/310496289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 0x56371e4fa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- 172.21.15.62:0/310496289 >> 172.21.15.62:0/310496289 conn(0x56371e520780 msgr2=0x56371e522b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- 172.21.15.62:0/310496289 shutdown_connections 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- 172.21.15.62:0/310496289 wait complete. 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- start start 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 Processor -- start 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4f9c80 0x56371e4f9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4fa5a0 0x56371e52aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e524f10 0x56371e52b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56371e3fee50 con 0x56371e4f9c80 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x56371e2ffa60 con 0x56371e524f10 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x56371e2fdcf0 con 0x56371e4fa5a0 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 10 client.0 _ll_register_callbacks cb 0x56371e4dfa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e89a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4f9c80 0x56371e4f9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e89a6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4f9c80 0x56371e4f9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56520/0 (socket says 172.21.15.62:56520) 2024-04-18T10:06:44.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e89a6640 1 -- 172.21.15.62:0/431579571 learned_addr learned my addr 172.21.15.62:0/431579571 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:44.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e3fff640 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4fa5a0 0x56371e52aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e91a7640 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e524f10 0x56371e52b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e89a6640 1 -- 172.21.15.62:0/431579571 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4fa5a0 msgr2=0x56371e52aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:44.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e89a6640 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4fa5a0 0x56371e52aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e89a6640 1 -- 172.21.15.62:0/431579571 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e524f10 msgr2=0x56371e52b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:44.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e89a6640 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e524f10 0x56371e52b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e89a6640 1 -- 172.21.15.62:0/431579571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56371e30de00 con 0x56371e4f9c80 2024-04-18T10:06:44.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e89a6640 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4f9c80 0x56371e4f9330 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f47d401a0a0 tx=0x7f47d40046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee080304e804e08 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:44.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-04-18T10:06:44.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47c37fe640 1 -- 172.21.15.62:0/431579571 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d4003d70 con 0x56371e4f9c80 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47c37fe640 1 -- 172.21.15.62:0/431579571 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f47d40c2460 con 0x56371e4f9c80 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- 172.21.15.62:0/431579571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47d805ec10 con 0x56371e4f9c80 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47c37fe640 1 -- 172.21.15.62:0/431579571 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f47d40cf050 con 0x56371e4f9c80 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 10 client.5097 fetch_fsmap learned FSMap version 16 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- 172.21.15.62:0/431579571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x56371e3c4670 con 0x56371e4f9c80 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47c37fe640 1 -- 172.21.15.62:0/431579571 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d805ec10 con 0x56371e4f9c80 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47c37fe640 1 -- 172.21.15.62:0/431579571 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 16) ==== 2284+0+0 (secure 0 0 0) 0x7f47d40d2020 con 0x56371e4f9c80 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 10 client.5097 fetch_fsmap finished waiting for FSMap version 16 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 10 client.5097 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 20 client.5097 populate_metadata read hostname 'smithi062' 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.62:6832/2971073685,v1:172.21.15.62:6833/2971073685] conn(0x56371e4f9870 0x56371e52ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 4 client.5097 mds_command: new command op to 4932 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-18T10:06:44.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47edd8c000 1 -- 172.21.15.62:0/431579571 --> [v2:172.21.15.62:6832/2971073685,v1:172.21.15.62:6833/2971073685] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56371e2dbbc0 con 0x56371e4f9870 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.489+0000 7f47e3fff640 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.62:6832/2971073685,v1:172.21.15.62:6833/2971073685] conn(0x56371e4f9870 0x56371e52ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47e3fff640 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.62:6832/2971073685,v1:172.21.15.62:6833/2971073685] conn(0x56371e4f9870 0x56371e52ba10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ef4773dcbaff5cc server_cookie=95c5d9d989adaa8b in_seq=0 out_seq=0 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47c37fe640 10 client.5097 ms_handle_connect on v2:172.21.15.62:6832/2971073685 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47c37fe640 1 -- 172.21.15.62:0/431579571 <== mds.0 v2:172.21.15.62:6832/2971073685 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56371e2dbbc0 con 0x56371e4f9870 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47c37fe640 10 client.5097 handle_command_reply: tid=0 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47edd8c000 1 client.5097 shutdown 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47edd8c000 1 -- 172.21.15.62:0/431579571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4f9c80 msgr2=0x56371e4f9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47edd8c000 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4f9c80 0x56371e4f9330 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f47d401a0a0 tx=0x7f47d40046d0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47edd8c000 1 -- 172.21.15.62:0/431579571 shutdown_connections 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47edd8c000 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.62:6832/2971073685,v1:172.21.15.62:6833/2971073685] conn(0x56371e4f9870 0x56371e52ba10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47edd8c000 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e524f10 0x56371e52b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47edd8c000 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4fa5a0 0x56371e52aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47c37fe640 0 client.5097 ms_handle_reset on v2:172.21.15.62:6832/2971073685 2024-04-18T10:06:44.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47edd8c000 1 --2- 172.21.15.62:0/431579571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e4f9c80 0x56371e4f9330 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47edd8c000 1 -- 172.21.15.62:0/431579571 >> 172.21.15.62:0/431579571 conn(0x56371e520780 msgr2=0x56371e522b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:44.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47edd8c000 1 -- 172.21.15.62:0/431579571 shutdown_connections 2024-04-18T10:06:44.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47edd8c000 1 -- 172.21.15.62:0/431579571 wait complete. 2024-04-18T10:06:44.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.493+0000 7f47edd8c000 20 client.5097 trim_cache size 0 max 16384 2024-04-18T10:06:44.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47edd8c000 1 Processor -- start 2024-04-18T10:06:44.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47edd8c000 1 -- start start 2024-04-18T10:06:44.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47edd8c000 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 0x56371e4f92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47edd8c000 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e4fa5a0 0x56371e5402b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47edd8c000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 0x56371e522b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47edd8c000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f47e41eac30 con 0x56371e524f10 2024-04-18T10:06:44.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47edd8c000 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f47e40b26d0 con 0x56371e4fa5a0 2024-04-18T10:06:44.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47edd8c000 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x56371e3417e0 con 0x56371e4f9c80 2024-04-18T10:06:44.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47e91a7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 0x56371e522b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.517 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47e91a7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 0x56371e522b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56532/0 (socket says 172.21.15.62:56532) 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47e91a7640 1 -- 172.21.15.62:0/568261302 learned_addr learned my addr 172.21.15.62:0/568261302 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47e91a7640 1 -- 172.21.15.62:0/568261302 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 msgr2=0x56371e4f92c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47e91a7640 1 --2- 172.21.15.62:0/568261302 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 0x56371e4f92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47e3fff640 1 --2- 172.21.15.62:0/568261302 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e4fa5a0 0x56371e5402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47e91a7640 1 -- 172.21.15.62:0/568261302 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e4fa5a0 msgr2=0x56371e5402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47e91a7640 1 --2- 172.21.15.62:0/568261302 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e4fa5a0 0x56371e5402b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.509+0000 7f47e91a7640 1 -- 172.21.15.62:0/568261302 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56371e39b340 con 0x56371e524f10 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e91a7640 1 --2- 172.21.15.62:0/568261302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 0x56371e522b70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f47dc0073a0 tx=0x7f47dc007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f744d8cbb4dadebe server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47c9ffb640 1 -- 172.21.15.62:0/568261302 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc010040 con 0x56371e524f10 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47c9ffb640 1 -- 172.21.15.62:0/568261302 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47dc01a070 con 0x56371e524f10 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47c9ffb640 1 -- 172.21.15.62:0/568261302 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc001d80 con 0x56371e524f10 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 -- 172.21.15.62:0/568261302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 msgr2=0x56371e522b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.518 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 --2- 172.21.15.62:0/568261302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 0x56371e522b70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f47dc0073a0 tx=0x7f47dc007860 comp rx=0 tx=0).stop 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 -- 172.21.15.62:0/568261302 shutdown_connections 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 --2- 172.21.15.62:0/568261302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 0x56371e522b70 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 --2- 172.21.15.62:0/568261302 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e4fa5a0 0x56371e5402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 --2- 172.21.15.62:0/568261302 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 0x56371e4f92c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 -- 172.21.15.62:0/568261302 >> 172.21.15.62:0/568261302 conn(0x56371e520780 msgr2=0x56371e52c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 -- 172.21.15.62:0/568261302 shutdown_connections 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 -- 172.21.15.62:0/568261302 wait complete. 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 -- start start 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 Processor -- start 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 0x56371e59d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e4fa5a0 0x56371e59de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 0x56371e56d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x56371e34f6e0 con 0x56371e524f10 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x56371e346b00 con 0x56371e4fa5a0 2024-04-18T10:06:44.519 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x56371e2d87b0 con 0x56371e4f9c80 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 10 client.0 _ll_register_callbacks cb 0x7f47d40ae070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e91a7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 0x56371e56d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e91a7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 0x56371e56d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56536/0 (socket says 172.21.15.62:56536) 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e91a7640 1 -- 172.21.15.62:0/833720112 learned_addr learned my addr 172.21.15.62:0/833720112 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e3fff640 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e4fa5a0 0x56371e59de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e89a6640 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 0x56371e59d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e91a7640 1 -- 172.21.15.62:0/833720112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 msgr2=0x56371e59d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e91a7640 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 0x56371e59d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e91a7640 1 -- 172.21.15.62:0/833720112 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e4fa5a0 msgr2=0x56371e59de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e91a7640 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e4fa5a0 0x56371e59de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e91a7640 1 -- 172.21.15.62:0/833720112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56371e2d47c0 con 0x56371e524f10 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e91a7640 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 0x56371e56d9e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x56371e2e1a00 tx=0x7f47dc000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37917f17844d94e server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47c37fe640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-04-18T10:06:44.520 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47c37fe640 1 -- 172.21.15.62:0/833720112 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc015920 con 0x56371e524f10 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 -- 172.21.15.62:0/833720112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47dc01a070 con 0x56371e524f10 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47c37fe640 1 -- 172.21.15.62:0/833720112 <== mon.0 v2:172.21.15.62:3300/0 2 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f47dc015ac0 con 0x56371e524f10 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47c37fe640 1 -- 172.21.15.62:0/833720112 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f47dc030050 con 0x56371e524f10 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 10 client.5103 fetch_fsmap learned FSMap version 16 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 -- 172.21.15.62:0/833720112 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x56371e2e7650 con 0x56371e524f10 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47c37fe640 1 -- 172.21.15.62:0/833720112 <== mon.0 v2:172.21.15.62:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47dc01a070 con 0x56371e524f10 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47c37fe640 1 -- 172.21.15.62:0/833720112 <== mon.0 v2:172.21.15.62:3300/0 5 ==== fsmap(e 16) ==== 2284+0+0 (secure 0 0 0) 0x7f47dc033020 con 0x56371e524f10 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 10 client.5103 fetch_fsmap finished waiting for FSMap version 16 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 10 client.5103 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 20 client.5103 populate_metadata read hostname 'smithi062' 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.62:6832/2971073685,v1:172.21.15.62:6833/2971073685] conn(0x56371e56ec20 0x56371e5a6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 4 client.5103 mds_command: new command op to 4932 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47edd8c000 1 -- 172.21.15.62:0/833720112 --> [v2:172.21.15.62:6832/2971073685,v1:172.21.15.62:6833/2971073685] -- command(tid 0: {"prefix": "status"}) -- 0x56371e3174e0 con 0x56371e56ec20 2024-04-18T10:06:44.521 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e89a6640 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.62:6832/2971073685,v1:172.21.15.62:6833/2971073685] conn(0x56371e56ec20 0x56371e5a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47c37fe640 10 client.5103 ms_handle_connect on v2:172.21.15.62:6832/2971073685 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47e89a6640 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.62:6832/2971073685,v1:172.21.15.62:6833/2971073685] conn(0x56371e56ec20 0x56371e5a6790 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfa7f7b71a53f773 server_cookie=713c76d5ac4b569c in_seq=0 out_seq=0 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.513+0000 7f47c37fe640 1 -- 172.21.15.62:0/833720112 <== mds.0 v2:172.21.15.62:6832/2971073685 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x56371e3174e0 con 0x56371e56ec20 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47c37fe640 10 client.5103 handle_command_reply: tid=0 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47edd8c000 1 client.5103 shutdown 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47edd8c000 1 -- 172.21.15.62:0/833720112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 msgr2=0x56371e56d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47edd8c000 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 0x56371e56d9e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x56371e2e1a00 tx=0x7f47dc000f50 comp rx=0 tx=0).stop 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47edd8c000 1 -- 172.21.15.62:0/833720112 shutdown_connections 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47edd8c000 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.62:6832/2971073685,v1:172.21.15.62:6833/2971073685] conn(0x56371e56ec20 0x56371e5a6790 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47edd8c000 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56371e524f10 0x56371e56d9e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47edd8c000 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56371e4fa5a0 0x56371e59de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47c37fe640 0 client.5103 ms_handle_reset on v2:172.21.15.62:6832/2971073685 2024-04-18T10:06:44.522 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47edd8c000 1 --2- 172.21.15.62:0/833720112 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56371e4f9c80 0x56371e59d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47edd8c000 1 -- 172.21.15.62:0/833720112 >> 172.21.15.62:0/833720112 conn(0x56371e520780 msgr2=0x56371e53f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:44.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47edd8c000 1 -- 172.21.15.62:0/833720112 shutdown_connections 2024-04-18T10:06:44.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47edd8c000 1 -- 172.21.15.62:0/833720112 wait complete. 2024-04-18T10:06:44.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47edd8c000 20 client.5103 trim_cache size 0 max 16384 2024-04-18T10:06:44.523 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:06:44.523 INFO:teuthology.orchestra.run.smithi062.stdout: "cluster_fsid": "26864971-6592-407b-b492-2c0e3b7ea567", 2024-04-18T10:06:44.523 INFO:teuthology.orchestra.run.smithi062.stdout: "whoami": 0, 2024-04-18T10:06:44.523 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 4932, 2024-04-18T10:06:44.523 INFO:teuthology.orchestra.run.smithi062.stdout: "want_state": "up:active", 2024-04-18T10:06:44.523 INFO:teuthology.orchestra.run.smithi062.stdout: "state": "up:active", 2024-04-18T10:06:44.523 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs", 2024-04-18T10:06:44.524 INFO:teuthology.orchestra.run.smithi062.stdout: "rank_uptime": 5.5805994370000001, 2024-04-18T10:06:44.524 INFO:teuthology.orchestra.run.smithi062.stdout: "mdsmap_epoch": 16, 2024-04-18T10:06:44.524 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch": 30, 2024-04-18T10:06:44.524 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_epoch_barrier": 30, 2024-04-18T10:06:44.524 INFO:teuthology.orchestra.run.smithi062.stdout: "uptime": 12.475384889000001 2024-04-18T10:06:44.524 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:06:44.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47eac0a640 1 -- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f47d00683b0 msgr2=0x7f47d006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47eac0a640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f47d00683b0 0x7f47d006a870 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f47d40ad000 tx=0x7f47d40a6000 comp rx=0 tx=0).stop 2024-04-18T10:06:44.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47eac0a640 1 -- 172.21.15.62:0/1106218201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47e409c4d0 msgr2=0x7f47e413e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47eac0a640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47e409c4d0 0x7f47e413e420 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f47d800c430 tx=0x7f47d800c8f0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47eac0a640 1 -- 172.21.15.62:0/1106218201 shutdown_connections 2024-04-18T10:06:44.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47eac0a640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f47d00683b0 0x7f47d006a870 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47eac0a640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e409e2d0 0x7f47e41e1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47eac0a640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f47e409c4d0 0x7f47e413e420 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47eac0a640 1 --2- 172.21.15.62:0/1106218201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e409b620 0x7f47e4071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.525 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47eac0a640 1 -- 172.21.15.62:0/1106218201 >> 172.21.15.62:0/1106218201 conn(0x7f47e40b9420 msgr2=0x7f47e40b7920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:44.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47eac0a640 1 -- 172.21.15.62:0/1106218201 shutdown_connections 2024-04-18T10:06:44.526 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.517+0000 7f47eac0a640 1 -- 172.21.15.62:0/1106218201 wait complete. 2024-04-18T10:06:44.531 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-18T10:06:44.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.633+0000 7fce08a1f640 1 Processor -- start 2024-04-18T10:06:44.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.633+0000 7fce08a1f640 1 -- start start 2024-04-18T10:06:44.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce08a1f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce0413ebd0 0x7fce040b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce08a1f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fce040b8db0 0x7fce040b9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce08a1f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fce04147f60 0x7fce041449b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce08a1f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fce040b68b0 con 0x7fce040b8db0 2024-04-18T10:06:44.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce08a1f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fce040b6280 con 0x7fce04147f60 2024-04-18T10:06:44.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce08a1f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fce040b6d40 con 0x7fce0413ebd0 2024-04-18T10:06:44.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce01d9b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fce040b8db0 0x7fce040b9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce01d9b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fce040b8db0 0x7fce040b9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56546/0 (socket says 172.21.15.62:56546) 2024-04-18T10:06:44.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce01d9b640 1 -- 172.21.15.62:0/394185917 learned_addr learned my addr 172.21.15.62:0/394185917 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:44.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce0259c640 1 --2- 172.21.15.62:0/394185917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce0413ebd0 0x7fce040b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce02d9d640 1 --2- 172.21.15.62:0/394185917 >> v2:172.21.15.145:3300/0 conn(0x7fce04147f60 0x7fce041449b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce01d9b640 1 -- 172.21.15.62:0/394185917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce0413ebd0 msgr2=0x7fce040b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:44.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce01d9b640 1 --2- 172.21.15.62:0/394185917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce0413ebd0 0x7fce040b82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce01d9b640 1 -- 172.21.15.62:0/394185917 >> v2:172.21.15.145:3300/0 conn(0x7fce04147f60 msgr2=0x7fce041449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:44.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce01d9b640 1 --2- 172.21.15.62:0/394185917 >> v2:172.21.15.145:3300/0 conn(0x7fce04147f60 0x7fce041449b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce01d9b640 1 -- 172.21.15.62:0/394185917 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce040b6540 con 0x7fce040b8db0 2024-04-18T10:06:44.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce01d9b640 1 --2- 172.21.15.62:0/394185917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce040b8db0 0x7fce040b9a60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fce04071490 tx=0x7fcdf401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c911fd0053d0e44 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:44.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce0159a640 1 -- 172.21.15.62:0/394185917 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf40b0070 con 0x7fce040b8db0 2024-04-18T10:06:44.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce0159a640 1 -- 172.21.15.62:0/394185917 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdf40ab600 con 0x7fce040b8db0 2024-04-18T10:06:44.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.637+0000 7fce0159a640 1 -- 172.21.15.62:0/394185917 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf40ab9d0 con 0x7fce040b8db0 2024-04-18T10:06:44.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 -- 172.21.15.62:0/394185917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce040b8db0 msgr2=0x7fce040b9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 --2- 172.21.15.62:0/394185917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce040b8db0 0x7fce040b9a60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fce04071490 tx=0x7fcdf401c490 comp rx=0 tx=0).stop 2024-04-18T10:06:44.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 -- 172.21.15.62:0/394185917 shutdown_connections 2024-04-18T10:06:44.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 --2- 172.21.15.62:0/394185917 >> v2:172.21.15.145:3300/0 conn(0x7fce04147f60 0x7fce041449b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 --2- 172.21.15.62:0/394185917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce040b8db0 0x7fce040b9a60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 --2- 172.21.15.62:0/394185917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce0413ebd0 0x7fce040b82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 -- 172.21.15.62:0/394185917 >> 172.21.15.62:0/394185917 conn(0x7fce0409c250 msgr2=0x7fce0409c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:44.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 -- 172.21.15.62:0/394185917 shutdown_connections 2024-04-18T10:06:44.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 -- 172.21.15.62:0/394185917 wait complete. 2024-04-18T10:06:44.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 Processor -- start 2024-04-18T10:06:44.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 -- start start 2024-04-18T10:06:44.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce040b8db0 0x7fce040b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce0413ebd0 0x7fce04143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce04147f60 0x7fce040b0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fce0406bc50 con 0x7fce040b8db0 2024-04-18T10:06:44.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fce0406bad0 con 0x7fce0413ebd0 2024-04-18T10:06:44.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fce0406bdd0 con 0x7fce04147f60 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce0259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce040b8db0 0x7fce040b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce0259c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce040b8db0 0x7fce040b2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56552/0 (socket says 172.21.15.62:56552) 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce0259c640 1 -- 172.21.15.62:0/40977699 learned_addr learned my addr 172.21.15.62:0/40977699 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce02d9d640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce04147f60 0x7fce040b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce01d9b640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce0413ebd0 0x7fce04143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce0259c640 1 -- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce04147f60 msgr2=0x7fce040b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce0259c640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce04147f60 0x7fce040b0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce0259c640 1 -- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce0413ebd0 msgr2=0x7fce04143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce0259c640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce0413ebd0 0x7fce04143bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce0259c640 1 -- 172.21.15.62:0/40977699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce040b4460 con 0x7fce040b8db0 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce01d9b640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce0413ebd0 0x7fce04143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce0259c640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce040b8db0 0x7fce040b2140 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800d800 tx=0x7fcdf800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fcdf2ffd640 1 -- 172.21.15.62:0/40977699 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8018020 con 0x7fce040b8db0 2024-04-18T10:06:44.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fcdf2ffd640 1 -- 172.21.15.62:0/40977699 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdf8009640 con 0x7fce040b8db0 2024-04-18T10:06:44.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fcdf2ffd640 1 -- 172.21.15.62:0/40977699 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdf80099f0 con 0x7fce040b8db0 2024-04-18T10:06:44.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 -- 172.21.15.62:0/40977699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce040b3270 con 0x7fce040b8db0 2024-04-18T10:06:44.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.641+0000 7fce08a1f640 1 -- 172.21.15.62:0/40977699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce040b1c20 con 0x7fce040b8db0 2024-04-18T10:06:44.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.645+0000 7fce08a1f640 1 -- 172.21.15.62:0/40977699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce04069230 con 0x7fce040b8db0 2024-04-18T10:06:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.645+0000 7fcdf2ffd640 1 -- 172.21.15.62:0/40977699 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcdf801b020 con 0x7fce040b8db0 2024-04-18T10:06:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.645+0000 7fcdf2ffd640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcde4068380 0x7fcde406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.645+0000 7fcdf2ffd640 1 -- 172.21.15.62:0/40977699 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fcdf8086ec0 con 0x7fce040b8db0 2024-04-18T10:06:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.645+0000 7fce01d9b640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcde4068380 0x7fcde406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:44.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.645+0000 7fce01d9b640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcde4068380 0x7fcde406a840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40ad000 tx=0x7fcdf4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:44.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.649+0000 7fcdf2ffd640 1 -- 172.21.15.62:0/40977699 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcdf8022020 con 0x7fce040b8db0 2024-04-18T10:06:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.837+0000 7fce08a1f640 1 -- 172.21.15.62:0/40977699 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fce041462c0 con 0x7fce040b8db0 2024-04-18T10:06:44.839 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fcdf2ffd640 1 -- 172.21.15.62:0/40977699 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7fcdf8057490 con 0x7fce040b8db0 2024-04-18T10:06:44.840 INFO:teuthology.orchestra.run.smithi062.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-18T10:06:44.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fce08a1f640 1 -- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcde4068380 msgr2=0x7fcde406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fce08a1f640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcde4068380 0x7fcde406a840 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40ad000 tx=0x7fcdf4008040 comp rx=0 tx=0).stop 2024-04-18T10:06:44.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fce08a1f640 1 -- 172.21.15.62:0/40977699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce040b8db0 msgr2=0x7fce040b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:44.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fce08a1f640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce040b8db0 0x7fce040b2140 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800d800 tx=0x7fcdf800dcc0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fce08a1f640 1 -- 172.21.15.62:0/40977699 shutdown_connections 2024-04-18T10:06:44.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fce08a1f640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcde4068380 0x7fcde406a840 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fce08a1f640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce04147f60 0x7fce040b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fce08a1f640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce0413ebd0 0x7fce04143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fce08a1f640 1 --2- 172.21.15.62:0/40977699 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce040b8db0 0x7fce040b2140 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:44.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fce08a1f640 1 -- 172.21.15.62:0/40977699 >> 172.21.15.62:0/40977699 conn(0x7fce0409c250 msgr2=0x7fce0406c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:44.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fce08a1f640 1 -- 172.21.15.62:0/40977699 shutdown_connections 2024-04-18T10:06:44.842 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:44.841+0000 7fce08a1f640 1 -- 172.21.15.62:0/40977699 wait complete. 2024-04-18T10:06:44.857 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-18T10:06:44.857 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-18T10:06:44.857 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi062.front.sepia.ceph.com 2024-04-18T10:06:44.857 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-18T10:06:44.857 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:44.857 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-18T10:06:44.857 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-18T10:06:44.857 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-18T10:06:44.857 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-18T10:06:44.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:44.858 DEBUG:teuthology.orchestra.run.smithi062:> ip addr 2024-04-18T10:06:44.864 INFO:teuthology.orchestra.run.smithi062.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-18T10:06:44.864 INFO:teuthology.orchestra.run.smithi062.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-18T10:06:44.864 INFO:teuthology.orchestra.run.smithi062.stdout: inet 127.0.0.1/8 scope host lo 2024-04-18T10:06:44.864 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-18T10:06:44.864 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 ::1/128 scope host 2024-04-18T10:06:44.864 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-18T10:06:44.864 INFO:teuthology.orchestra.run.smithi062.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-18T10:06:44.864 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:14 brd ff:ff:ff:ff:ff:ff 2024-04-18T10:06:44.864 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f0 2024-04-18T10:06:44.864 INFO:teuthology.orchestra.run.smithi062.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:a9:aa:15 brd ff:ff:ff:ff:ff:ff 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout: altname enp7s0f1 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c2 brd ff:ff:ff:ff:ff:ff 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0c:c4:7a:bb:d1:c3 brd ff:ff:ff:ff:ff:ff 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.21.15.62/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft 42470sec preferred_lft 42470sec 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 02:42:4b:60:3b:63 brd ff:ff:ff:ff:ff:ff 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-18T10:06:44.865 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-18T10:06:44.866 INFO:teuthology.orchestra.run.smithi062.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-18T10:06:44.866 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-18T10:06:44.866 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-18T10:06:44.866 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-18T10:06:44.866 INFO:teuthology.orchestra.run.smithi062.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-18T10:06:44.866 INFO:teuthology.orchestra.run.smithi062.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:44.866 INFO:teuthology.orchestra.run.smithi062.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-18T10:06:44.866 INFO:teuthology.orchestra.run.smithi062.stdout: valid_lft forever preferred_lft forever 2024-04-18T10:06:44.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:44.866 DEBUG:teuthology.orchestra.run.smithi062:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-18T10:06:44.918 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-04-18T10:06:44.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:44.919 DEBUG:teuthology.orchestra.run.smithi062:> route 2024-04-18T10:06:44.976 INFO:teuthology.orchestra.run.smithi062.stdout:Kernel IP routing table 2024-04-18T10:06:44.976 INFO:teuthology.orchestra.run.smithi062.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-18T10:06:44.976 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-18T10:06:44.976 INFO:teuthology.orchestra.run.smithi062.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-18T10:06:44.977 INFO:teuthology.orchestra.run.smithi062.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-18T10:06:44.977 INFO:teuthology.orchestra.run.smithi062.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-18T10:06:44.977 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-18T10:06:44.977 INFO:teuthology.orchestra.run.smithi062.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-18T10:06:44.977 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-18T10:06:44.977 INFO:teuthology.orchestra.run.smithi062.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-18T10:06:44.977 INFO:teuthology.orchestra.run.smithi062.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-18T10:06:44.977 DEBUG:teuthology.orchestra.run.smithi062:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-18T10:06:44.977 DEBUG:teuthology.orchestra.run.smithi062:> set -e 2024-04-18T10:06:44.977 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-18T10:06:44.977 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-18T10:06:44.977 DEBUG:teuthology.orchestra.run.smithi062:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-18T10:06:44.977 DEBUG:teuthology.orchestra.run.smithi062:> ') 2024-04-18T10:06:45.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-18T10:06:45.091 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-18T10:06:45.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:45.099 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-04-18T10:06:45.104 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-18T10:06:45.105 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-18T10:06:45.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:45.105 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link set brx.0 up 2024-04-18T10:06:45.158 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-18T10:06:45.158 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-18T10:06:45.158 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:45.206 INFO:teuthology.orchestra.run.smithi062.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-18T10:06:45.206 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-18T10:06:45.207 DEBUG:teuthology.orchestra.run.smithi062:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:45.252 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-04-18T10:06:45.310 DEBUG:teuthology.orchestra.run.smithi062:> cat /proc/self/mounts | awk '{print $2}' 2024-04-18T10:06:45.357 INFO:teuthology.orchestra.run.smithi062.stdout:/sys 2024-04-18T10:06:45.357 INFO:teuthology.orchestra.run.smithi062.stdout:/proc 2024-04-18T10:06:45.357 INFO:teuthology.orchestra.run.smithi062.stdout:/dev 2024-04-18T10:06:45.357 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/pts 2024-04-18T10:06:45.357 INFO:teuthology.orchestra.run.smithi062.stdout:/run 2024-04-18T10:06:45.357 INFO:teuthology.orchestra.run.smithi062.stdout:/ 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/security 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/shm 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/run/lock 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/cgroup 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/pstore 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/bpf 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/hugepages 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/mqueue 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/debug 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/tracing 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/fs/nfsd 2024-04-18T10:06:45.358 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/fuse/connections 2024-04-18T10:06:45.359 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/config 2024-04-18T10:06:45.359 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-sysusers.service 2024-04-18T10:06:45.359 INFO:teuthology.orchestra.run.smithi062.stdout:/run/rpc_pipefs 2024-04-18T10:06:45.359 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/1000 2024-04-18T10:06:45.359 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph 2024-04-18T10:06:45.359 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-0 2024-04-18T10:06:45.359 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-1 2024-04-18T10:06:45.359 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-2 2024-04-18T10:06:45.359 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-3 2024-04-18T10:06:45.359 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns 2024-04-18T10:06:45.359 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:45.359 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:45.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:45.360 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-04-18T10:06:45.408 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-18T10:06:45.409 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-18T10:06:45.453 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-04-18T10:06:45.467 DEBUG:teuthology.orchestra.run.smithi062:> cat /proc/self/mounts | awk '{print $2}' 2024-04-18T10:06:45.516 INFO:teuthology.orchestra.run.smithi062.stdout:/sys 2024-04-18T10:06:45.516 INFO:teuthology.orchestra.run.smithi062.stdout:/proc 2024-04-18T10:06:45.516 INFO:teuthology.orchestra.run.smithi062.stdout:/dev 2024-04-18T10:06:45.516 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/pts 2024-04-18T10:06:45.516 INFO:teuthology.orchestra.run.smithi062.stdout:/run 2024-04-18T10:06:45.516 INFO:teuthology.orchestra.run.smithi062.stdout:/ 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/security 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/shm 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/run/lock 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/cgroup 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/pstore 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/bpf 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/hugepages 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/mqueue 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/debug 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/tracing 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/fs/nfsd 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/fuse/connections 2024-04-18T10:06:45.517 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/config 2024-04-18T10:06:45.518 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-sysusers.service 2024-04-18T10:06:45.518 INFO:teuthology.orchestra.run.smithi062.stdout:/run/rpc_pipefs 2024-04-18T10:06:45.518 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/1000 2024-04-18T10:06:45.518 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph 2024-04-18T10:06:45.518 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-0 2024-04-18T10:06:45.518 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-1 2024-04-18T10:06:45.518 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-2 2024-04-18T10:06:45.518 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-3 2024-04-18T10:06:45.518 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns 2024-04-18T10:06:45.518 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:45.518 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:45.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:45.519 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-04-18T10:06:45.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:2024-04-18T10:06:45.537+0000 7f7358b9a580 -1 init, newargv = 0x559aff1bab90 newargc=15 2024-04-18T10:06:45.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[36646]: starting ceph client 2024-04-18T10:06:45.542 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[36646]: starting fuse 2024-04-18T10:06:46.643 DEBUG:teuthology.orchestra.run.smithi062:> sudo modprobe fuse 2024-04-18T10:06:46.660 DEBUG:teuthology.orchestra.run.smithi062:> cat /proc/self/mounts | awk '{print $2}' 2024-04-18T10:06:46.711 INFO:teuthology.orchestra.run.smithi062.stdout:/sys 2024-04-18T10:06:46.711 INFO:teuthology.orchestra.run.smithi062.stdout:/proc 2024-04-18T10:06:46.711 INFO:teuthology.orchestra.run.smithi062.stdout:/dev 2024-04-18T10:06:46.711 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/pts 2024-04-18T10:06:46.711 INFO:teuthology.orchestra.run.smithi062.stdout:/run 2024-04-18T10:06:46.711 INFO:teuthology.orchestra.run.smithi062.stdout:/ 2024-04-18T10:06:46.711 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/security 2024-04-18T10:06:46.711 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/shm 2024-04-18T10:06:46.711 INFO:teuthology.orchestra.run.smithi062.stdout:/run/lock 2024-04-18T10:06:46.711 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/cgroup 2024-04-18T10:06:46.711 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/pstore 2024-04-18T10:06:46.711 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/bpf 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/sys/fs/binfmt_misc 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/hugepages 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/dev/mqueue 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/debug 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/tracing 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/proc/fs/nfsd 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/fs/fuse/connections 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/sys/kernel/config 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/run/credentials/systemd-sysusers.service 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/run/rpc_pipefs 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/run/user/1000 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-0 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-1 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-2 2024-04-18T10:06:46.712 INFO:teuthology.orchestra.run.smithi062.stdout:/var/lib/ceph/osd/ceph-3 2024-04-18T10:06:46.713 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns 2024-04-18T10:06:46.713 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:46.713 INFO:teuthology.orchestra.run.smithi062.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:46.713 INFO:teuthology.orchestra.run.smithi062.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:46.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:46.714 DEBUG:teuthology.orchestra.run.smithi062:> ls /sys/fs/fuse/connections 2024-04-18T10:06:46.760 INFO:teuthology.orchestra.run.smithi062.stdout:45 2024-04-18T10:06:46.761 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-18T10:06:46.761 DEBUG:teuthology.orchestra.run.smithi062:> sudo stdin-killer -- python3 -c ' 2024-04-18T10:06:46.761 DEBUG:teuthology.orchestra.run.smithi062:> import glob 2024-04-18T10:06:46.761 DEBUG:teuthology.orchestra.run.smithi062:> import re 2024-04-18T10:06:46.761 DEBUG:teuthology.orchestra.run.smithi062:> import os 2024-04-18T10:06:46.761 DEBUG:teuthology.orchestra.run.smithi062:> import subprocess 2024-04-18T10:06:46.761 DEBUG:teuthology.orchestra.run.smithi062:> 2024-04-18T10:06:46.761 DEBUG:teuthology.orchestra.run.smithi062:> def _find_admin_socket(client_name): 2024-04-18T10:06:46.761 DEBUG:teuthology.orchestra.run.smithi062:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-18T10:06:46.761 DEBUG:teuthology.orchestra.run.smithi062:> files = glob.glob(asok_path) 2024-04-18T10:06:46.761 DEBUG:teuthology.orchestra.run.smithi062:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> # Given a non-glob path, it better be there 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> if "*" not in asok_path: 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> assert(len(files) == 1) 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> return files[0] 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> for f in files: 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> contents = proc_f.read() 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> if mountpoint in contents: 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> return f 2024-04-18T10:06:46.762 DEBUG:teuthology.orchestra.run.smithi062:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-18T10:06:46.763 DEBUG:teuthology.orchestra.run.smithi062:> 2024-04-18T10:06:46.763 DEBUG:teuthology.orchestra.run.smithi062:> print(_find_admin_socket("client.0")) 2024-04-18T10:06:46.763 DEBUG:teuthology.orchestra.run.smithi062:> ' 2024-04-18T10:06:46.888 INFO:teuthology.orchestra.run.smithi062.stdout:/var/run/ceph/ceph-client.0.36646.asok 2024-04-18T10:06:46.892 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-18T10:06:46.901 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.36646.asok 2024-04-18T10:06:46.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:46.901 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.36646.asok status 2024-04-18T10:06:47.064 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:06:47.064 INFO:teuthology.orchestra.run.smithi062.stdout: "metadata": { 2024-04-18T10:06:47.064 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_sha1": "a6549451253d26216d353246c0fe6f6d80f3b9cf", 2024-04-18T10:06:47.064 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 19.0.0-3115-ga6549451 (a6549451253d26216d353246c0fe6f6d80f3b9cf) squid (dev)", 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "entity_id": "0", 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "hostname": "smithi062", 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "pid": "36646", 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "root": "/" 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_count": 0, 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "dentry_pinned_count": 0, 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "id": 5115, 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "inst": { 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "name": { 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "client", 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "num": 5115 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:47.065 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 906856079 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": { 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: "type": "v1", 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: "addr": "192.168.0.1:0", 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: "nonce": 906856079 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: "inst_str": "client.5115 192.168.0.1:0/906856079", 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: "addr_str": "192.168.0.1:0/906856079", 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: "inode_count": 1, 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_epoch": 16, 2024-04-18T10:06:47.066 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch": 30, 2024-04-18T10:06:47.067 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_epoch_barrier": 0, 2024-04-18T10:06:47.067 INFO:teuthology.orchestra.run.smithi062.stdout: "blocklisted": false, 2024-04-18T10:06:47.067 INFO:teuthology.orchestra.run.smithi062.stdout: "fs_name": "cephfs" 2024-04-18T10:06:47.067 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:06:47.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:47.078 DEBUG:teuthology.orchestra.run.smithi062:> stat --file-system '--printf=%T 2024-04-18T10:06:47.078 DEBUG:teuthology.orchestra.run.smithi062:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:47.085 INFO:teuthology.orchestra.run.smithi062.stdout:fuseblk 2024-04-18T10:06:47.085 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:47.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:47.085 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:47.139 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-04-18T10:06:47.192 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.d9VoKJKw9n 2024-04-18T10:06:47.192 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:06:47.192 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/tmp/tmp.d9VoKJKw9n 2024-04-18T10:06:47.244 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'mkdir -p dir1/dir2/dir3') 2024-04-18T10:06:47.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-18T10:06:47.359 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-18T10:06:47.379 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'chmod 755 /tmp/test-cephfs86v2e119') 2024-04-18T10:06:47.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:47 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-18T10:06:47.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-18T10:06:47.438 INFO:tasks.cephfs.test_cephfs_shell:Running script "/tmp/test-cephfs86v2e119" 2024-04-18T10:06:47.438 DEBUG:teuthology.orchestra.run.smithi062:> cephfs-shell -b /tmp/test-cephfs86v2e119 2024-04-18T10:06:47.694 INFO:teuthology.orchestra.run.smithi062.stderr:Unrecognized command 2024-04-18T10:06:47.720 DEBUG:teuthology.orchestra.run:got remote process result: 127 2024-04-18T10:06:47.721 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-18T10:06:47.721 DEBUG:teuthology.orchestra.run.smithi062:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-18T10:06:47.781 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-18T10:06:47.825 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-18T10:06:47.826 INFO:tasks.cephfs.mount:Terminating background process 2024-04-18T10:06:47.826 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:06:47.826 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-18T10:06:47.832 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:48.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e17f0a640 1 Processor -- start 2024-04-18T10:06:48.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e17f0a640 1 -- start start 2024-04-18T10:06:48.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e17f0a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e100b9d90 0x7f9e101450e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e17f0a640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9e100b4940 0x7f9e10148840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e17f0a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e10145620 con 0x7f9e100b44e0 2024-04-18T10:06:48.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e17f0a640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f9e100b4dd0 con 0x7f9e100b4940 2024-04-18T10:06:48.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e17f0a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f9e10145b80 con 0x7f9e100b9d90 2024-04-18T10:06:48.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e154a5640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9e100b44e0 0x7f9e10148130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42910/0 (socket says 172.21.15.62:42910) 2024-04-18T10:06:48.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e154a5640 1 -- 172.21.15.62:0/2375208110 learned_addr learned my addr 172.21.15.62:0/2375208110 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:48.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e15ca6640 1 --2- 172.21.15.62:0/2375208110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e100b9d90 0x7f9e101450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e164a7640 1 --2- 172.21.15.62:0/2375208110 >> v2:172.21.15.145:3300/0 conn(0x7f9e100b4940 0x7f9e10148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1819391969 0 0) 0x7f9e00001120 con 0x7f9e100b44e0 2024-04-18T10:06:48.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9dfc005740 con 0x7f9e100b44e0 2024-04-18T10:06:48.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3934426236 0 0) 0x7f9dfc005740 con 0x7f9e100b44e0 2024-04-18T10:06:48.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e100b9d90 msgr2=0x7f9e101450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:48.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 --2- 172.21.15.62:0/2375208110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e100b9d90 0x7f9e101450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 >> v2:172.21.15.145:3300/0 conn(0x7f9e100b4940 msgr2=0x7f9e10148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:48.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 --2- 172.21.15.62:0/2375208110 >> v2:172.21.15.145:3300/0 conn(0x7f9e100b4940 0x7f9e10148840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e100ba170 con 0x7f9e100b44e0 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f9e00001850 con 0x7f9e100b44e0 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 >> v1:172.21.15.62:6789/0 conn(0x7f9e100b44e0 legacy=0x7f9e10148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 --2- 172.21.15.62:0/2375208110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e100b9d90 0x7f9dfc005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 --2- 172.21.15.62:0/2375208110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dfc006180 0x7f9dfc006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 --2- 172.21.15.62:0/2375208110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dfc006f60 0x7f9dfc0073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e10145b80 con 0x7f9dfc006f60 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f9dfc00dbe0 con 0x7f9dfc006180 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f9dfc00dde0 con 0x7f9e100b9d90 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e164a7640 1 --2- 172.21.15.62:0/2375208110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dfc006f60 0x7f9dfc0073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e154a5640 1 --2- 172.21.15.62:0/2375208110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dfc006180 0x7f9dfc006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e15ca6640 1 --2- 172.21.15.62:0/2375208110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e100b9d90 0x7f9dfc005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e164a7640 1 -- 172.21.15.62:0/2375208110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e100b9d90 msgr2=0x7f9dfc005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e164a7640 1 --2- 172.21.15.62:0/2375208110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e100b9d90 0x7f9dfc005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e164a7640 1 -- 172.21.15.62:0/2375208110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dfc006180 msgr2=0x7f9dfc006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e164a7640 1 --2- 172.21.15.62:0/2375208110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9dfc006180 0x7f9dfc006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.009+0000 7f9e164a7640 1 -- 172.21.15.62:0/2375208110 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dfc005740 con 0x7f9dfc006f60 2024-04-18T10:06:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e164a7640 1 --2- 172.21.15.62:0/2375208110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dfc006f60 0x7f9dfc0073e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009fc0 tx=0x7f9e0801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126b6a674a800cf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e080b2070 con 0x7f9dfc006f60 2024-04-18T10:06:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e080a5cd0 con 0x7f9dfc006f60 2024-04-18T10:06:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e14ca4640 1 -- 172.21.15.62:0/2375208110 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e08009c10 con 0x7f9dfc006f60 2024-04-18T10:06:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 -- 172.21.15.62:0/2375208110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dfc006f60 msgr2=0x7f9dfc0073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 --2- 172.21.15.62:0/2375208110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9dfc006f60 0x7f9dfc0073e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9e08009fc0 tx=0x7f9e0801ccc0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e164a7640 1 -- 172.21.15.62:0/2375208110 reap_dead start 2024-04-18T10:06:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 -- 172.21.15.62:0/2375208110 shutdown_connections 2024-04-18T10:06:48.014 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 -- 172.21.15.62:0/2375208110 >> 172.21.15.62:0/2375208110 conn(0x7f9e1006c0d0 msgr2=0x7f9e100b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 -- 172.21.15.62:0/2375208110 shutdown_connections 2024-04-18T10:06:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 -- 172.21.15.62:0/2375208110 wait complete. 2024-04-18T10:06:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 Processor -- start 2024-04-18T10:06:48.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 -- start start 2024-04-18T10:06:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dfc006f60 0x7f9e101ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e101dd090 0x7f9e101dd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e10143f00 0x7f9e101443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9e100b5b20 con 0x7f9e10143f00 2024-04-18T10:06:48.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f9e100b3fc0 con 0x7f9e101dd090 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e17f0a640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f9e100b5ca0 con 0x7f9dfc006f60 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e164a7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e10143f00 0x7f9e101443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e164a7640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e10143f00 0x7f9e101443b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56586/0 (socket says 172.21.15.62:56586) 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e164a7640 1 -- 172.21.15.62:0/1558204006 learned_addr learned my addr 172.21.15.62:0/1558204006 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e15ca6640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dfc006f60 0x7f9e101ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e164a7640 1 -- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dfc006f60 msgr2=0x7f9e101ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e164a7640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dfc006f60 0x7f9e101ce250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e164a7640 1 -- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e101dd090 msgr2=0x7f9e101dd510 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e164a7640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e101dd090 0x7f9e101dd510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.013+0000 7f9e164a7640 1 -- 172.21.15.62:0/1558204006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e101d25a0 con 0x7f9e10143f00 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.017+0000 7f9e164a7640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e10143f00 0x7f9e101443b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9e080b0040 tx=0x7f9e08009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.017+0000 7f9e067fc640 1 -- 172.21.15.62:0/1558204006 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e080bd070 con 0x7f9e10143f00 2024-04-18T10:06:48.017 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.017+0000 7f9e067fc640 1 -- 172.21.15.62:0/1558204006 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e080a5ad0 con 0x7f9e10143f00 2024-04-18T10:06:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.017+0000 7f9e067fc640 1 -- 172.21.15.62:0/1558204006 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e080b2040 con 0x7f9e10143f00 2024-04-18T10:06:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.017+0000 7f9e17f0a640 1 -- 172.21.15.62:0/1558204006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e10071a90 con 0x7f9e10143f00 2024-04-18T10:06:48.018 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.017+0000 7f9e17f0a640 1 -- 172.21.15.62:0/1558204006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e101d1da0 con 0x7f9e10143f00 2024-04-18T10:06:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.017+0000 7f9e17f0a640 1 -- 172.21.15.62:0/1558204006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e10068b90 con 0x7f9e10143f00 2024-04-18T10:06:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.017+0000 7f9e067fc640 1 -- 172.21.15.62:0/1558204006 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9e080039a0 con 0x7f9e10143f00 2024-04-18T10:06:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.017+0000 7f9e067fc640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9df40683b0 0x7f9df406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.017+0000 7f9e067fc640 1 -- 172.21.15.62:0/1558204006 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f9e081207a0 con 0x7f9e10143f00 2024-04-18T10:06:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.017+0000 7f9e15ca6640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9df40683b0 0x7f9df406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.017+0000 7f9e15ca6640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9df40683b0 0x7f9df406a870 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9e100b2df0 tx=0x7f9e0c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:48.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.021+0000 7f9e067fc640 1 -- 172.21.15.62:0/1558204006 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9e080169d0 con 0x7f9e10143f00 2024-04-18T10:06:48.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.185+0000 7f9e17f0a640 1 -- 172.21.15.62:0/1558204006 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e100717b0 con 0x7f9e10143f00 2024-04-18T10:06:48.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.189+0000 7f9e067fc640 1 -- 172.21.15.62:0/1558204006 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16657 (secure 0 0 0) 0x7f9e080a5ca0 con 0x7f9e10143f00 2024-04-18T10:06:48.190 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:48.191 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":30,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:39.937405+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:06:36.802689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-18T10:06:37.269237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:48.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.193+0000 7f9e17f0a640 1 -- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9df40683b0 msgr2=0x7f9df406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.193+0000 7f9e17f0a640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9df40683b0 0x7f9df406a870 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9e100b2df0 tx=0x7f9e0c008040 comp rx=0 tx=0).stop 2024-04-18T10:06:48.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.193+0000 7f9e17f0a640 1 -- 172.21.15.62:0/1558204006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e10143f00 msgr2=0x7f9e101443b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.193+0000 7f9e17f0a640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e10143f00 0x7f9e101443b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9e080b0040 tx=0x7f9e08009750 comp rx=0 tx=0).stop 2024-04-18T10:06:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.193+0000 7f9e17f0a640 1 -- 172.21.15.62:0/1558204006 shutdown_connections 2024-04-18T10:06:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.193+0000 7f9e17f0a640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9df40683b0 0x7f9df406a870 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.193+0000 7f9e17f0a640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9e10143f00 0x7f9e101443b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.193+0000 7f9e17f0a640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e101dd090 0x7f9e101dd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.193+0000 7f9e17f0a640 1 --2- 172.21.15.62:0/1558204006 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9dfc006f60 0x7f9e101ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.193+0000 7f9e17f0a640 1 -- 172.21.15.62:0/1558204006 >> 172.21.15.62:0/1558204006 conn(0x7f9e1006c0d0 msgr2=0x7f9e100b6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.193+0000 7f9e17f0a640 1 -- 172.21.15.62:0/1558204006 shutdown_connections 2024-04-18T10:06:48.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.193+0000 7f9e17f0a640 1 -- 172.21.15.62:0/1558204006 wait complete. 2024-04-18T10:06:48.210 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-18T10:06:36.802689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-18T10:06:37.269237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:48.211 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:48.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71b1431640 1 Processor -- start 2024-04-18T10:06:48.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71b1431640 1 -- start start 2024-04-18T10:06:48.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71b1431640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f71ac06d500 0x7f71ac0aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71b1431640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71ac0af330 0x7f71ac09aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71b1431640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71ac09af90 0x7f71ac09b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71b1431640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f71ac06e510 con 0x7f71ac0af330 2024-04-18T10:06:48.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71b1431640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f71ac0ae680 con 0x7f71ac06d500 2024-04-18T10:06:48.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71b1431640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f71ac06df00 con 0x7f71ac09af90 2024-04-18T10:06:48.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71aa7fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71ac0af330 0x7f71ac09aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71aa7fc640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f71ac0af330 0x7f71ac09aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56602/0 (socket says 172.21.15.62:56602) 2024-04-18T10:06:48.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71aa7fc640 1 -- 172.21.15.62:0/1801634732 learned_addr learned my addr 172.21.15.62:0/1801634732 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:48.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71aaffd640 1 --2- 172.21.15.62:0/1801634732 >> v2:172.21.15.145:3300/0 conn(0x7f71ac06d500 0x7f71ac0aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71ab7fe640 1 --2- 172.21.15.62:0/1801634732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71ac09af90 0x7f71ac09b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71aa7fc640 1 -- 172.21.15.62:0/1801634732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71ac09af90 msgr2=0x7f71ac09b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:48.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71aa7fc640 1 --2- 172.21.15.62:0/1801634732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71ac09af90 0x7f71ac09b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71aa7fc640 1 -- 172.21.15.62:0/1801634732 >> v2:172.21.15.145:3300/0 conn(0x7f71ac06d500 msgr2=0x7f71ac0aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:48.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71aa7fc640 1 --2- 172.21.15.62:0/1801634732 >> v2:172.21.15.145:3300/0 conn(0x7f71ac06d500 0x7f71ac0aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.333+0000 7f71aa7fc640 1 -- 172.21.15.62:0/1801634732 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71ac06d8e0 con 0x7f71ac0af330 2024-04-18T10:06:48.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71aa7fc640 1 --2- 172.21.15.62:0/1801634732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71ac0af330 0x7f71ac09aa50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f71ac06cb20 tx=0x7f719401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e83107fc35ed78 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:48.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71a9ffb640 1 -- 172.21.15.62:0/1801634732 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71940b2070 con 0x7f71ac0af330 2024-04-18T10:06:48.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71a9ffb640 1 -- 172.21.15.62:0/1801634732 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71940ad640 con 0x7f71ac0af330 2024-04-18T10:06:48.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71a9ffb640 1 -- 172.21.15.62:0/1801634732 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71940ada10 con 0x7f71ac0af330 2024-04-18T10:06:48.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 -- 172.21.15.62:0/1801634732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71ac0af330 msgr2=0x7f71ac09aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 --2- 172.21.15.62:0/1801634732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71ac0af330 0x7f71ac09aa50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f71ac06cb20 tx=0x7f719401c3d0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 -- 172.21.15.62:0/1801634732 shutdown_connections 2024-04-18T10:06:48.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 --2- 172.21.15.62:0/1801634732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71ac09af90 0x7f71ac09b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 --2- 172.21.15.62:0/1801634732 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71ac0af330 0x7f71ac09aa50 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 --2- 172.21.15.62:0/1801634732 >> v2:172.21.15.145:3300/0 conn(0x7f71ac06d500 0x7f71ac0aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 -- 172.21.15.62:0/1801634732 >> 172.21.15.62:0/1801634732 conn(0x7f71ac0b9690 msgr2=0x7f71ac0b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:48.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 -- 172.21.15.62:0/1801634732 shutdown_connections 2024-04-18T10:06:48.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 -- 172.21.15.62:0/1801634732 wait complete. 2024-04-18T10:06:48.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 Processor -- start 2024-04-18T10:06:48.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 -- start start 2024-04-18T10:06:48.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71ac06d500 0x7f71ac14ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71ac09af90 0x7f71ac1e4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71ac0af330 0x7f71ac1e5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac0b5630 con 0x7f71ac06d500 2024-04-18T10:06:48.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac0b57b0 con 0x7f71ac09af90 2024-04-18T10:06:48.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71b1431640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f71ac0b5a30 con 0x7f71ac0af330 2024-04-18T10:06:48.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71aaffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71ac06d500 0x7f71ac14ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71aaffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71ac06d500 0x7f71ac14ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56608/0 (socket says 172.21.15.62:56608) 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71aaffd640 1 -- 172.21.15.62:0/1324976810 learned_addr learned my addr 172.21.15.62:0/1324976810 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71ab7fe640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71ac0af330 0x7f71ac1e5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71aa7fc640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71ac09af90 0x7f71ac1e4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71aaffd640 1 -- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71ac0af330 msgr2=0x7f71ac1e5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71aaffd640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71ac0af330 0x7f71ac1e5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71aaffd640 1 -- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71ac09af90 msgr2=0x7f71ac1e4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71aaffd640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71ac09af90 0x7f71ac1e4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71aaffd640 1 -- 172.21.15.62:0/1324976810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71ac1e8620 con 0x7f71ac06d500 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71aa7fc640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71ac09af90 0x7f71ac1e4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.337+0000 7f71aaffd640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71ac06d500 0x7f71ac14ad20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f719c00efc0 tx=0x7f719c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.341+0000 7f71a37fe640 1 -- 172.21.15.62:0/1324976810 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c018020 con 0x7f71ac06d500 2024-04-18T10:06:48.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.341+0000 7f71a37fe640 1 -- 172.21.15.62:0/1324976810 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f719c014480 con 0x7f71ac06d500 2024-04-18T10:06:48.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.341+0000 7f71b1431640 1 -- 172.21.15.62:0/1324976810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71ac1e8900 con 0x7f71ac06d500 2024-04-18T10:06:48.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.341+0000 7f71a37fe640 1 -- 172.21.15.62:0/1324976810 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c014850 con 0x7f71ac06d500 2024-04-18T10:06:48.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.341+0000 7f71b1431640 1 -- 172.21.15.62:0/1324976810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71ac09cbe0 con 0x7f71ac06d500 2024-04-18T10:06:48.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.341+0000 7f71b1431640 1 -- 172.21.15.62:0/1324976810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71ac068c00 con 0x7f71ac06d500 2024-04-18T10:06:48.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.341+0000 7f71a37fe640 1 -- 172.21.15.62:0/1324976810 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f719c01b020 con 0x7f71ac06d500 2024-04-18T10:06:48.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.341+0000 7f71a37fe640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f718c068380 0x7f718c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.341+0000 7f71a37fe640 1 -- 172.21.15.62:0/1324976810 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f719c0874c0 con 0x7f71ac06d500 2024-04-18T10:06:48.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.341+0000 7f71aa7fc640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f718c068380 0x7f718c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.343 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.341+0000 7f71aa7fc640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f718c068380 0x7f718c06a840 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f71940af000 tx=0x7f71940a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:48.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.345+0000 7f71a37fe640 1 -- 172.21.15.62:0/1324976810 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f719c022070 con 0x7f71ac06d500 2024-04-18T10:06:48.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.505+0000 7f71b1431640 1 -- 172.21.15.62:0/1324976810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f71ac0695f0 con 0x7f71ac06d500 2024-04-18T10:06:48.507 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.505+0000 7f71a37fe640 1 -- 172.21.15.62:0/1324976810 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) ==== 93+0+10 (secure 0 0 0) 0x7f719c010040 con 0x7f71ac06d500 2024-04-18T10:06:48.507 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:48.511 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.509+0000 7f71b1431640 1 -- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f718c068380 msgr2=0x7f718c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.509+0000 7f71b1431640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f718c068380 0x7f718c06a840 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f71940af000 tx=0x7f71940a7070 comp rx=0 tx=0).stop 2024-04-18T10:06:48.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.509+0000 7f71b1431640 1 -- 172.21.15.62:0/1324976810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71ac06d500 msgr2=0x7f71ac14ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.509+0000 7f71b1431640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71ac06d500 0x7f71ac14ad20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f719c00efc0 tx=0x7f719c00c490 comp rx=0 tx=0).stop 2024-04-18T10:06:48.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.509+0000 7f71b1431640 1 -- 172.21.15.62:0/1324976810 shutdown_connections 2024-04-18T10:06:48.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.509+0000 7f71b1431640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f718c068380 0x7f718c06a840 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.509+0000 7f71b1431640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71ac0af330 0x7f71ac1e5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.509+0000 7f71b1431640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71ac09af90 0x7f71ac1e4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.509+0000 7f71b1431640 1 --2- 172.21.15.62:0/1324976810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f71ac06d500 0x7f71ac14ad20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.509+0000 7f71b1431640 1 -- 172.21.15.62:0/1324976810 >> 172.21.15.62:0/1324976810 conn(0x7f71ac0b9690 msgr2=0x7f71ac14d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:48.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.513+0000 7f71b1431640 1 -- 172.21.15.62:0/1324976810 shutdown_connections 2024-04-18T10:06:48.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.513+0000 7f71b1431640 1 -- 172.21.15.62:0/1324976810 wait complete. 2024-04-18T10:06:48.528 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-18T10:06:48.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.649+0000 7fa1c4b14640 1 Processor -- start 2024-04-18T10:06:48.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1c4b14640 1 -- start start 2024-04-18T10:06:48.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1c4b14640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa1c0144950 0x7fa1c00b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1c4b14640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1c006e170 con 0x7fa1c006dd90 2024-04-18T10:06:48.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1c4b14640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1c00af850 con 0x7fa1c0147f50 2024-04-18T10:06:48.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1c4b14640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fa1c00af620 con 0x7fa1c0144950 2024-04-18T10:06:48.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1bdd9b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa1c0144950 0x7fa1c00b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1be59c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa1c006dd90 0x7fa1c0147840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42912/0 (socket says 172.21.15.62:42912) 2024-04-18T10:06:48.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1be59c640 1 -- 172.21.15.62:0/3469885607 learned_addr learned my addr 172.21.15.62:0/3469885607 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:48.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1bdd9b640 1 -- 172.21.15.62:0/3469885607 >> v1:172.21.15.145:6789/0 conn(0x7fa1c0147f50 legacy=0x7fa1c014eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:48.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1bd59a640 1 -- 172.21.15.62:0/3469885607 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 419901880 0 0) 0x7fa1c006e170 con 0x7fa1c006dd90 2024-04-18T10:06:48.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1bdd9b640 1 -- 172.21.15.62:0/3469885607 >> v1:172.21.15.62:6789/0 conn(0x7fa1c006dd90 legacy=0x7fa1c0147840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1bdd9b640 1 -- 172.21.15.62:0/3469885607 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c00afb30 con 0x7fa1c0144950 2024-04-18T10:06:48.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.653+0000 7fa1bdd9b640 1 --2- 172.21.15.62:0/3469885607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1c0144950 0x7fa1c00b43a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40097c0 tx=0x7fa1b401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9432fc696de9568e server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:48.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bd59a640 1 -- 172.21.15.62:0/3469885607 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b40b0070 con 0x7fa1c0144950 2024-04-18T10:06:48.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bd59a640 1 -- 172.21.15.62:0/3469885607 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b40ab600 con 0x7fa1c0144950 2024-04-18T10:06:48.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bd59a640 1 -- 172.21.15.62:0/3469885607 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b40ab9d0 con 0x7fa1c0144950 2024-04-18T10:06:48.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3469885607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1c0144950 msgr2=0x7fa1c00b43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 --2- 172.21.15.62:0/3469885607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1c0144950 0x7fa1c00b43a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40097c0 tx=0x7fa1b401c410 comp rx=0 tx=0).stop 2024-04-18T10:06:48.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3469885607 shutdown_connections 2024-04-18T10:06:48.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 --2- 172.21.15.62:0/3469885607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1c0144950 0x7fa1c00b43a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3469885607 >> 172.21.15.62:0/3469885607 conn(0x7fa1c00b9690 msgr2=0x7fa1c00b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:48.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3469885607 shutdown_connections 2024-04-18T10:06:48.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3469885607 wait complete. 2024-04-18T10:06:48.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 Processor -- start 2024-04-18T10:06:48.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 -- start start 2024-04-18T10:06:48.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1c006dd90 0x7fa1c013c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1c0144950 0x7fa1c013f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1c0147f50 0x7fa1c013fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c0153540 con 0x7fa1c0144950 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c01533c0 con 0x7fa1c006dd90 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c01536c0 con 0x7fa1c0147f50 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bdd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1c0144950 0x7fa1c013f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bdd9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1c0144950 0x7fa1c013f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56618/0 (socket says 172.21.15.62:56618) 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bdd9b640 1 -- 172.21.15.62:0/3748400282 learned_addr learned my addr 172.21.15.62:0/3748400282 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1be59c640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1c006dd90 0x7fa1c013c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bed9d640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1c0147f50 0x7fa1c013fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bdd9b640 1 -- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1c0147f50 msgr2=0x7fa1c013fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bdd9b640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1c0147f50 0x7fa1c013fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bdd9b640 1 -- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1c006dd90 msgr2=0x7fa1c013c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bdd9b640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1c006dd90 0x7fa1c013c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bdd9b640 1 -- 172.21.15.62:0/3748400282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c013bb20 con 0x7fa1c0144950 2024-04-18T10:06:48.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1be59c640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1c006dd90 0x7fa1c013c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:48.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bed9d640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1c0147f50 0x7fa1c013fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:48.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1bdd9b640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1c0144950 0x7fa1c013f7f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40097c0 tx=0x7fa1b401ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:48.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1aeffd640 1 -- 172.21.15.62:0/3748400282 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b40b0050 con 0x7fa1c0144950 2024-04-18T10:06:48.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1aeffd640 1 -- 172.21.15.62:0/3748400282 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b40b1040 con 0x7fa1c0144950 2024-04-18T10:06:48.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1aeffd640 1 -- 172.21.15.62:0/3748400282 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4004260 con 0x7fa1c0144950 2024-04-18T10:06:48.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3748400282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c009d480 con 0x7fa1c0144950 2024-04-18T10:06:48.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.657+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3748400282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c013e050 con 0x7fa1c0144950 2024-04-18T10:06:48.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.661+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3748400282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c0068c00 con 0x7fa1c0144950 2024-04-18T10:06:48.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.661+0000 7fa1aeffd640 1 -- 172.21.15.62:0/3748400282 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa1b40be060 con 0x7fa1c0144950 2024-04-18T10:06:48.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.661+0000 7fa1aeffd640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa1a00683b0 0x7fa1a006a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.661+0000 7fa1aeffd640 1 -- 172.21.15.62:0/3748400282 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fa1b4120730 con 0x7fa1c0144950 2024-04-18T10:06:48.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.661+0000 7fa1be59c640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa1a00683b0 0x7fa1a006a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.661+0000 7fa1be59c640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa1a00683b0 0x7fa1a006a870 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0007fd0 tx=0x7fa1b0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:48.665 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.665+0000 7fa1aeffd640 1 -- 172.21.15.62:0/3748400282 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa1b40a4e20 con 0x7fa1c0144950 2024-04-18T10:06:48.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.825+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3748400282 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa1c00b0670 con 0x7fa1c0144950 2024-04-18T10:06:48.826 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.825+0000 7fa1aeffd640 1 -- 172.21.15.62:0/3748400282 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v30) ==== 104+0+11 (secure 0 0 0) 0x7fa1b411ee20 con 0x7fa1c0144950 2024-04-18T10:06:48.826 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-18T10:06:48.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.829+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa1a00683b0 msgr2=0x7fa1a006a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.829+0000 7fa1c4b14640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa1a00683b0 0x7fa1a006a870 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0007fd0 tx=0x7fa1b0009040 comp rx=0 tx=0).stop 2024-04-18T10:06:48.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.829+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3748400282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1c0144950 msgr2=0x7fa1c013f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.829+0000 7fa1c4b14640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1c0144950 0x7fa1c013f7f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40097c0 tx=0x7fa1b401ac50 comp rx=0 tx=0).stop 2024-04-18T10:06:48.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.829+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3748400282 shutdown_connections 2024-04-18T10:06:48.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.829+0000 7fa1c4b14640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa1a00683b0 0x7fa1a006a870 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.829+0000 7fa1c4b14640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1c0147f50 0x7fa1c013fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.829+0000 7fa1c4b14640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1c0144950 0x7fa1c013f7f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.829+0000 7fa1c4b14640 1 --2- 172.21.15.62:0/3748400282 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1c006dd90 0x7fa1c013c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.829+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3748400282 >> 172.21.15.62:0/3748400282 conn(0x7fa1c00b9690 msgr2=0x7fa1c00b4970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:48.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.829+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3748400282 shutdown_connections 2024-04-18T10:06:48.829 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.829+0000 7fa1c4b14640 1 -- 172.21.15.62:0/3748400282 wait complete. 2024-04-18T10:06:48.844 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-18T10:06:48.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f13968e3640 1 Processor -- start 2024-04-18T10:06:48.960 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f13968e3640 1 -- start start 2024-04-18T10:06:48.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f13968e3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f13900b4c90 0x7f1390145f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f13968e3640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f139006e490 0x7f13901464c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f13968e3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f139006d4f0 con 0x7f139009e090 2024-04-18T10:06:48.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f13968e3640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f139006d2a0 con 0x7f139006e490 2024-04-18T10:06:48.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f13968e3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f139006d7d0 con 0x7f13900b4c90 2024-04-18T10:06:48.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f138f7fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f139006e490 0x7f13901464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f1394e80640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f139009e090 0x7f139009e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42924/0 (socket says 172.21.15.62:42924) 2024-04-18T10:06:48.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f1394e80640 1 -- 172.21.15.62:0/621219455 learned_addr learned my addr 172.21.15.62:0/621219455 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:48.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f138ffff640 1 --2- 172.21.15.62:0/621219455 >> v2:172.21.15.145:3300/0 conn(0x7f13900b4c90 0x7f1390145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f138f7fe640 1 -- 172.21.15.62:0/621219455 >> v2:172.21.15.145:3300/0 conn(0x7f13900b4c90 msgr2=0x7f1390145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:48.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f138f7fe640 1 --2- 172.21.15.62:0/621219455 >> v2:172.21.15.145:3300/0 conn(0x7f13900b4c90 0x7f1390145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f138f7fe640 1 -- 172.21.15.62:0/621219455 >> v1:172.21.15.62:6789/0 conn(0x7f139009e090 legacy=0x7f139009e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.961+0000 7f138f7fe640 1 -- 172.21.15.62:0/621219455 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139009c570 con 0x7f139006e490 2024-04-18T10:06:48.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f138f7fe640 1 --2- 172.21.15.62:0/621219455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f139006e490 0x7f13901464c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f139009bff0 tx=0x7f138401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ed310ce0abe90e server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:48.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f138effd640 1 -- 172.21.15.62:0/621219455 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13840b0070 con 0x7f139006e490 2024-04-18T10:06:48.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f138effd640 1 -- 172.21.15.62:0/621219455 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13840ab600 con 0x7f139006e490 2024-04-18T10:06:48.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f138effd640 1 -- 172.21.15.62:0/621219455 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13840ab9d0 con 0x7f139006e490 2024-04-18T10:06:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 -- 172.21.15.62:0/621219455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f139006e490 msgr2=0x7f13901464c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 --2- 172.21.15.62:0/621219455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f139006e490 0x7f13901464c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f139009bff0 tx=0x7f138401c3f0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 -- 172.21.15.62:0/621219455 shutdown_connections 2024-04-18T10:06:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 --2- 172.21.15.62:0/621219455 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f139006e490 0x7f13901464c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 --2- 172.21.15.62:0/621219455 >> v2:172.21.15.145:3300/0 conn(0x7f13900b4c90 0x7f1390145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 -- 172.21.15.62:0/621219455 >> 172.21.15.62:0/621219455 conn(0x7f13900b9690 msgr2=0x7f13900b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 -- 172.21.15.62:0/621219455 shutdown_connections 2024-04-18T10:06:48.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 -- 172.21.15.62:0/621219455 wait complete. 2024-04-18T10:06:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 Processor -- start 2024-04-18T10:06:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 -- start start 2024-04-18T10:06:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f139006e490 0x7f139013ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f139009e090 0x7f13901e0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13900b4c90 0x7f13901e0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1390152fc0 con 0x7f13900b4c90 2024-04-18T10:06:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f1390152e40 con 0x7f139009e090 2024-04-18T10:06:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f13968e3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f1390153140 con 0x7f139006e490 2024-04-18T10:06:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f1394e80640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13900b4c90 0x7f13901e0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f1394e80640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13900b4c90 0x7f13901e0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56642/0 (socket says 172.21.15.62:56642) 2024-04-18T10:06:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f1394e80640 1 -- 172.21.15.62:0/391447573 learned_addr learned my addr 172.21.15.62:0/391447573 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:48.968 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f138ffff640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f139006e490 0x7f139013ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f138f7fe640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f139009e090 0x7f13901e0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f1394e80640 1 -- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f139006e490 msgr2=0x7f139013ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f1394e80640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f139006e490 0x7f139013ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f1394e80640 1 -- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f139009e090 msgr2=0x7f13901e0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f1394e80640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f139009e090 0x7f13901e0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f1394e80640 1 -- 172.21.15.62:0/391447573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13900ae9c0 con 0x7f13900b4c90 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f138ffff640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f139006e490 0x7f139013ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f138f7fe640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f139009e090 0x7f13901e0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.965+0000 7f1394e80640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13900b4c90 0x7f13901e0dd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f138800cbf0 tx=0x7f138800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.969+0000 7f138cff9640 1 -- 172.21.15.62:0/391447573 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388017070 con 0x7f13900b4c90 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.969+0000 7f138cff9640 1 -- 172.21.15.62:0/391447573 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f138800ad40 con 0x7f13900b4c90 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.969+0000 7f13968e3640 1 -- 172.21.15.62:0/391447573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13900b0240 con 0x7f13900b4c90 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.969+0000 7f138cff9640 1 -- 172.21.15.62:0/391447573 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388012600 con 0x7f13900b4c90 2024-04-18T10:06:48.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.969+0000 7f13968e3640 1 -- 172.21.15.62:0/391447573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13900aef20 con 0x7f13900b4c90 2024-04-18T10:06:48.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.969+0000 7f13968e3640 1 -- 172.21.15.62:0/391447573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1390068c00 con 0x7f13900b4c90 2024-04-18T10:06:48.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.969+0000 7f138cff9640 1 -- 172.21.15.62:0/391447573 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f138801a070 con 0x7f13900b4c90 2024-04-18T10:06:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.969+0000 7f138cff9640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f137c068380 0x7f137c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.969+0000 7f138cff9640 1 -- 172.21.15.62:0/391447573 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f1388086620 con 0x7f13900b4c90 2024-04-18T10:06:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.969+0000 7f138ffff640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f137c068380 0x7f137c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:48.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.969+0000 7f138ffff640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f137c068380 0x7f137c06a840 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1390069730 tx=0x7f1380008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:48.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:48.973+0000 7f138cff9640 1 -- 172.21.15.62:0/391447573 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f13900ae9c0 con 0x7f13900b4c90 2024-04-18T10:06:49.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.129+0000 7f13968e3640 1 -- 172.21.15.62:0/391447573 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1390139320 con 0x7f13900b4c90 2024-04-18T10:06:49.131 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.129+0000 7f138cff9640 1 -- 172.21.15.62:0/391447573 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v30) ==== 100+0+11 (secure 0 0 0) 0x7f1388056580 con 0x7f13900b4c90 2024-04-18T10:06:49.131 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-04-18T10:06:49.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.133+0000 7f13968e3640 1 -- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f137c068380 msgr2=0x7f137c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.133+0000 7f13968e3640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f137c068380 0x7f137c06a840 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1390069730 tx=0x7f1380008040 comp rx=0 tx=0).stop 2024-04-18T10:06:49.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.133+0000 7f13968e3640 1 -- 172.21.15.62:0/391447573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13900b4c90 msgr2=0x7f13901e0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.133+0000 7f13968e3640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13900b4c90 0x7f13901e0dd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f138800cbf0 tx=0x7f138800a430 comp rx=0 tx=0).stop 2024-04-18T10:06:49.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.133+0000 7f13968e3640 1 -- 172.21.15.62:0/391447573 shutdown_connections 2024-04-18T10:06:49.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.133+0000 7f13968e3640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f137c068380 0x7f137c06a840 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.133+0000 7f13968e3640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f13900b4c90 0x7f13901e0dd0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.133+0000 7f13968e3640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f139009e090 0x7f13901e0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.133+0000 7f13968e3640 1 --2- 172.21.15.62:0/391447573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f139006e490 0x7f139013ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.133+0000 7f13968e3640 1 -- 172.21.15.62:0/391447573 >> 172.21.15.62:0/391447573 conn(0x7f13900b9690 msgr2=0x7f13900b5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:49.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.133+0000 7f13968e3640 1 -- 172.21.15.62:0/391447573 shutdown_connections 2024-04-18T10:06:49.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.133+0000 7f13968e3640 1 -- 172.21.15.62:0/391447573 wait complete. 2024-04-18T10:06:49.149 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-18T10:06:49.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd3010640 1 Processor -- start 2024-04-18T10:06:49.265 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd3010640 1 -- start start 2024-04-18T10:06:49.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd3010640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3fcc0b7870 0x7f3fcc073860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:49.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd3010640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fcc147e30 0x7f3fcc1527b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:49.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd3010640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fcc148e10 con 0x7f3fcc073da0 2024-04-18T10:06:49.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd3010640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f3fcc148980 con 0x7f3fcc0b7870 2024-04-18T10:06:49.266 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd3010640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3fcc0b7c50 con 0x7f3fcc147e30 2024-04-18T10:06:49.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd0dac640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3fcc0b7870 0x7f3fcc073860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:49.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd0dac640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f3fcc0b7870 0x7f3fcc073860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56650/0 (socket says 172.21.15.62:56650) 2024-04-18T10:06:49.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd0dac640 1 -- 172.21.15.62:0/1632978576 learned_addr learned my addr 172.21.15.62:0/1632978576 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:49.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd15ad640 1 --2- 172.21.15.62:0/1632978576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fcc147e30 0x7f3fcc1527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:49.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd0dac640 1 -- 172.21.15.62:0/1632978576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fcc147e30 msgr2=0x7f3fcc1527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:49.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd0dac640 1 --2- 172.21.15.62:0/1632978576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fcc147e30 0x7f3fcc1527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd0dac640 1 -- 172.21.15.62:0/1632978576 >> v1:172.21.15.145:6789/0 conn(0x7f3fcc073da0 legacy=0x7f3fcc0b6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.265+0000 7f3fd0dac640 1 -- 172.21.15.62:0/1632978576 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fcc148330 con 0x7f3fcc0b7870 2024-04-18T10:06:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fd0dac640 1 --2- 172.21.15.62:0/1632978576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fcc0b7870 0x7f3fcc073860 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc009870 tx=0x7f3fbc01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6dd45373664835f server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:49.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fcb7fe640 1 -- 172.21.15.62:0/1632978576 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc0b0070 con 0x7f3fcc0b7870 2024-04-18T10:06:49.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fcb7fe640 1 -- 172.21.15.62:0/1632978576 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fbc0ab600 con 0x7f3fcc0b7870 2024-04-18T10:06:49.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fcb7fe640 1 -- 172.21.15.62:0/1632978576 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc0ab9d0 con 0x7f3fcc0b7870 2024-04-18T10:06:49.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fd3010640 1 -- 172.21.15.62:0/1632978576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fcc0b7870 msgr2=0x7f3fcc073860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fd3010640 1 --2- 172.21.15.62:0/1632978576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fcc0b7870 0x7f3fcc073860 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc009870 tx=0x7f3fbc01c440 comp rx=0 tx=0).stop 2024-04-18T10:06:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fd3010640 1 -- 172.21.15.62:0/1632978576 shutdown_connections 2024-04-18T10:06:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fd3010640 1 --2- 172.21.15.62:0/1632978576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fcc147e30 0x7f3fcc1527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fd3010640 1 --2- 172.21.15.62:0/1632978576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fcc0b7870 0x7f3fcc073860 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fd3010640 1 -- 172.21.15.62:0/1632978576 >> 172.21.15.62:0/1632978576 conn(0x7f3fcc0bb4d0 msgr2=0x7f3fcc0bb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fd3010640 1 -- 172.21.15.62:0/1632978576 shutdown_connections 2024-04-18T10:06:49.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fd3010640 1 -- 172.21.15.62:0/1632978576 wait complete. 2024-04-18T10:06:49.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fd3010640 1 Processor -- start 2024-04-18T10:06:49.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.269+0000 7f3fd3010640 1 -- start start 2024-04-18T10:06:49.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd3010640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fcc073da0 0x7f3fcc141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:49.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd3010640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fcc0b7870 0x7f3fcc1430c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:49.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd3010640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fcc147e30 0x7f3fcc143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd3010640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc155510 con 0x7f3fcc147e30 2024-04-18T10:06:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd3010640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc06cdf0 con 0x7f3fcc0b7870 2024-04-18T10:06:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd3010640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3fcc155690 con 0x7f3fcc073da0 2024-04-18T10:06:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd15ad640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fcc147e30 0x7f3fcc143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd15ad640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fcc147e30 0x7f3fcc143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56658/0 (socket says 172.21.15.62:56658) 2024-04-18T10:06:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd15ad640 1 -- 172.21.15.62:0/3269403458 learned_addr learned my addr 172.21.15.62:0/3269403458 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fcbfff640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fcc0b7870 0x7f3fcc1430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd0dac640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fcc073da0 0x7f3fcc141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd15ad640 1 -- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fcc073da0 msgr2=0x7f3fcc141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd15ad640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fcc073da0 0x7f3fcc141000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd15ad640 1 -- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fcc0b7870 msgr2=0x7f3fcc1430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd15ad640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fcc0b7870 0x7f3fcc1430c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd15ad640 1 -- 172.21.15.62:0/3269403458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fcc0ac960 con 0x7f3fcc147e30 2024-04-18T10:06:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd15ad640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fcc147e30 0x7f3fcc143600 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4007cf0 tx=0x7f3fc400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fc97fa640 1 -- 172.21.15.62:0/3269403458 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4014420 con 0x7f3fcc147e30 2024-04-18T10:06:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fc97fa640 1 -- 172.21.15.62:0/3269403458 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc4019070 con 0x7f3fcc147e30 2024-04-18T10:06:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd3010640 1 -- 172.21.15.62:0/3269403458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fcc0acc40 con 0x7f3fcc147e30 2024-04-18T10:06:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fc97fa640 1 -- 172.21.15.62:0/3269403458 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4014820 con 0x7f3fcc147e30 2024-04-18T10:06:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd3010640 1 -- 172.21.15.62:0/3269403458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fcc1428c0 con 0x7f3fcc147e30 2024-04-18T10:06:49.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd3010640 1 -- 172.21.15.62:0/3269403458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fcc068b20 con 0x7f3fcc147e30 2024-04-18T10:06:49.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fc97fa640 1 -- 172.21.15.62:0/3269403458 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3fc40149c0 con 0x7f3fcc147e30 2024-04-18T10:06:49.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fc97fa640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3fb8068380 0x7f3fb806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:49.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fc97fa640 1 -- 172.21.15.62:0/3269403458 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f3fc4087130 con 0x7f3fcc147e30 2024-04-18T10:06:49.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.273+0000 7f3fd0dac640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3fb8068380 0x7f3fb806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:49.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.277+0000 7f3fd0dac640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3fb8068380 0x7f3fb806a840 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc0ad000 tx=0x7f3fbc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:49.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.277+0000 7f3fc97fa640 1 -- 172.21.15.62:0/3269403458 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3fc4057780 con 0x7f3fcc147e30 2024-04-18T10:06:49.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.445+0000 7f3fd3010640 1 -- 172.21.15.62:0/3269403458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3fcc072f60 con 0x7f3fcc147e30 2024-04-18T10:06:49.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fc97fa640 1 -- 172.21.15.62:0/3269403458 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v30) ==== 80+0+0 (secure 0 0 0) 0x7f3fc4057080 con 0x7f3fcc147e30 2024-04-18T10:06:49.448 INFO:teuthology.orchestra.run.smithi062.stderr:listed 0 entries 2024-04-18T10:06:49.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fd3010640 1 -- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3fb8068380 msgr2=0x7f3fb806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fd3010640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3fb8068380 0x7f3fb806a840 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc0ad000 tx=0x7f3fbc008040 comp rx=0 tx=0).stop 2024-04-18T10:06:49.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fd3010640 1 -- 172.21.15.62:0/3269403458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fcc147e30 msgr2=0x7f3fcc143600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fd3010640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fcc147e30 0x7f3fcc143600 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3fc4007cf0 tx=0x7f3fc400e4b0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fd3010640 1 -- 172.21.15.62:0/3269403458 shutdown_connections 2024-04-18T10:06:49.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fd3010640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3fb8068380 0x7f3fb806a840 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fd3010640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3fcc147e30 0x7f3fcc143600 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fd3010640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fcc0b7870 0x7f3fcc1430c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fd3010640 1 --2- 172.21.15.62:0/3269403458 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fcc073da0 0x7f3fcc141000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fd3010640 1 -- 172.21.15.62:0/3269403458 >> 172.21.15.62:0/3269403458 conn(0x7f3fcc0bb4d0 msgr2=0x7f3fcc072600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:49.451 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fd3010640 1 -- 172.21.15.62:0/3269403458 shutdown_connections 2024-04-18T10:06:49.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.449+0000 7f3fd3010640 1 -- 172.21.15.62:0/3269403458 wait complete. 2024-04-18T10:06:49.465 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi062.front.sepia.ceph.com... 2024-04-18T10:06:49.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:49.466 DEBUG:teuthology.orchestra.run.smithi062:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-18T10:06:49.476 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi062.stderr:ceph-fuse[36646]: fuse finished with error 0 and tester_r 0 2024-04-18T10:06:49.478 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T10:06:49.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:49.478 DEBUG:teuthology.orchestra.run.smithi062:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-18T10:06:49.528 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:49.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe6eb9640 1 Processor -- start 2024-04-18T10:06:49.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe6eb9640 1 -- start start 2024-04-18T10:06:49.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe6eb9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fe00bbe40 0x7f8fe0141910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:49.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe6eb9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fe014be20 0x7f8fe01526c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:49.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe6eb9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fe00bc220 con 0x7f8fe00b7b50 2024-04-18T10:06:49.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe6eb9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f8fe00b5dd0 con 0x7f8fe00bbe40 2024-04-18T10:06:49.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe6eb9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8fe00b60b0 con 0x7f8fe014be20 2024-04-18T10:06:49.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe4c55640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fe00bbe40 0x7f8fe0141910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:49.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe4c55640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f8fe00bbe40 0x7f8fe0141910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56668/0 (socket says 172.21.15.62:56668) 2024-04-18T10:06:49.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe4c55640 1 -- 172.21.15.62:0/3447578899 learned_addr learned my addr 172.21.15.62:0/3447578899 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:49.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe5456640 1 --2- 172.21.15.62:0/3447578899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fe014be20 0x7f8fe01526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:49.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe4c55640 1 -- 172.21.15.62:0/3447578899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fe014be20 msgr2=0x7f8fe01526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:49.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe4c55640 1 --2- 172.21.15.62:0/3447578899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fe014be20 0x7f8fe01526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe4c55640 1 -- 172.21.15.62:0/3447578899 >> v1:172.21.15.145:6789/0 conn(0x7f8fe00b7b50 legacy=0x7f8fe014b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:49.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe4c55640 1 -- 172.21.15.62:0/3447578899 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe00b6390 con 0x7f8fe00bbe40 2024-04-18T10:06:49.695 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.693+0000 7f8fe5456640 1 --2- 172.21.15.62:0/3447578899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fe014be20 0x7f8fe01526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:49.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe4c55640 1 --2- 172.21.15.62:0/3447578899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00bbe40 0x7f8fe0141910 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009e50 tx=0x7f8fd001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122d411069609491 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:49.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdf7fe640 1 -- 172.21.15.62:0/3447578899 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd00a56f0 con 0x7f8fe00bbe40 2024-04-18T10:06:49.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdf7fe640 1 -- 172.21.15.62:0/3447578899 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd00a5890 con 0x7f8fe00bbe40 2024-04-18T10:06:49.696 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdf7fe640 1 -- 172.21.15.62:0/3447578899 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd00a5c60 con 0x7f8fe00bbe40 2024-04-18T10:06:49.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3447578899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00bbe40 msgr2=0x7f8fe0141910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 --2- 172.21.15.62:0/3447578899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00bbe40 0x7f8fe0141910 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8fd0009e50 tx=0x7f8fd001c960 comp rx=0 tx=0).stop 2024-04-18T10:06:49.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3447578899 shutdown_connections 2024-04-18T10:06:49.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 --2- 172.21.15.62:0/3447578899 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fe014be20 0x7f8fe01526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 --2- 172.21.15.62:0/3447578899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00bbe40 0x7f8fe0141910 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3447578899 >> 172.21.15.62:0/3447578899 conn(0x7f8fe0067e10 msgr2=0x7f8fe00bb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:49.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3447578899 shutdown_connections 2024-04-18T10:06:49.697 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3447578899 wait complete. 2024-04-18T10:06:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 Processor -- start 2024-04-18T10:06:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 -- start start 2024-04-18T10:06:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fe00b7b50 0x7f8fe01e40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:49.699 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00bbe40 0x7f8fe0145030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fe014be20 0x7f8fe014a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01557b0 con 0x7f8fe00bbe40 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0155630 con 0x7f8fe00b7b50 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8fe0155930 con 0x7f8fe014be20 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00bbe40 0x7f8fe0145030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00bbe40 0x7f8fe0145030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56670/0 (socket says 172.21.15.62:56670) 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdffff640 1 -- 172.21.15.62:0/3183549292 learned_addr learned my addr 172.21.15.62:0/3183549292 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe5456640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fe014be20 0x7f8fe014a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe4c55640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fe00b7b50 0x7f8fe01e40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdffff640 1 -- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fe014be20 msgr2=0x7f8fe014a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdffff640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fe014be20 0x7f8fe014a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdffff640 1 -- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fe00b7b50 msgr2=0x7f8fe01e40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdffff640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fe00b7b50 0x7f8fe01e40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdffff640 1 -- 172.21.15.62:0/3183549292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe00b3b60 con 0x7f8fe00bbe40 2024-04-18T10:06:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdffff640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00bbe40 0x7f8fe0145030 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400c450 tx=0x7f8fd400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdd7fa640 1 -- 172.21.15.62:0/3183549292 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4012830 con 0x7f8fe00bbe40 2024-04-18T10:06:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdd7fa640 1 -- 172.21.15.62:0/3183549292 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd4017070 con 0x7f8fe00bbe40 2024-04-18T10:06:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3183549292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe00b3e40 con 0x7f8fe00bbe40 2024-04-18T10:06:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fdd7fa640 1 -- 172.21.15.62:0/3183549292 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4012c30 con 0x7f8fe00bbe40 2024-04-18T10:06:49.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.697+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3183549292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe00b2a70 con 0x7f8fe00bbe40 2024-04-18T10:06:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.701+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3183549292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe0143f80 con 0x7f8fe00bbe40 2024-04-18T10:06:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.701+0000 7f8fdd7fa640 1 -- 172.21.15.62:0/3183549292 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8fd401c460 con 0x7f8fe00bbe40 2024-04-18T10:06:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.701+0000 7f8fdd7fa640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8fcc068380 0x7f8fcc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.701+0000 7f8fdd7fa640 1 -- 172.21.15.62:0/3183549292 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f8fd4086850 con 0x7f8fe00bbe40 2024-04-18T10:06:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.701+0000 7f8fe4c55640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8fcc068380 0x7f8fcc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:49.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.701+0000 7f8fe4c55640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8fcc068380 0x7f8fcc06a840 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00ae000 tx=0x7f8fd00a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:49.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.705+0000 7f8fdd7fa640 1 -- 172.21.15.62:0/3183549292 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8fe0143f80 con 0x7f8fe00bbe40 2024-04-18T10:06:49.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.905+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3183549292 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8fe0069230 con 0x7f8fe00bbe40 2024-04-18T10:06:49.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.905+0000 7f8fdd7fa640 1 -- 172.21.15.62:0/3183549292 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+3534 (secure 0 0 0) 0x7f8fd4056d70 con 0x7f8fe00bbe40 2024-04-18T10:06:49.906 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:49.906 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4705,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1649467687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1649467687},{"type":"v1","addr":"172.21.15.145:6835","nonce":1649467687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:06:38.304481+0000","modified":"2024-04-18T10:06:41.339980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4932":{"gid":4932,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2971073685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2971073685},{"type":"v1","addr":"172.21.15.62:6833","nonce":2971073685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4932,"qdb_cluster":[4932]},"id":2}]} 2024-04-18T10:06:49.906 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-04-18T10:06:49.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.905+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8fcc068380 msgr2=0x7f8fcc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.909+0000 7f8fe6eb9640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8fcc068380 0x7f8fcc06a840 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00ae000 tx=0x7f8fd00a7070 comp rx=0 tx=0).stop 2024-04-18T10:06:49.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.909+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3183549292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00bbe40 msgr2=0x7f8fe0145030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:49.908 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.909+0000 7f8fe6eb9640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00bbe40 0x7f8fe0145030 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f8fd400c450 tx=0x7f8fd400c910 comp rx=0 tx=0).stop 2024-04-18T10:06:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.909+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3183549292 shutdown_connections 2024-04-18T10:06:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.909+0000 7f8fe6eb9640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8fcc068380 0x7f8fcc06a840 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.909+0000 7f8fe6eb9640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8fe014be20 0x7f8fe014a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.909+0000 7f8fe6eb9640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8fe00bbe40 0x7f8fe0145030 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.909+0000 7f8fe6eb9640 1 --2- 172.21.15.62:0/3183549292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8fe00b7b50 0x7f8fe01e40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.909+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3183549292 >> 172.21.15.62:0/3183549292 conn(0x7f8fe0067e10 msgr2=0x7f8fe0147db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.909+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3183549292 shutdown_connections 2024-04-18T10:06:49.909 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:49.909+0000 7f8fe6eb9640 1 -- 172.21.15.62:0/3183549292 wait complete. 2024-04-18T10:06:49.924 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:50.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.049+0000 7f6e4c2df640 1 Processor -- start 2024-04-18T10:06:50.050 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.049+0000 7f6e4c2df640 1 -- start start 2024-04-18T10:06:50.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.049+0000 7f6e4c2df640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6e44148490 0x7f6e44148920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.049+0000 7f6e4c2df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e440b63b0 0x7f6e440b6820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.049+0000 7f6e4c2df640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e440b6d60 con 0x7f6e4406bd30 2024-04-18T10:06:50.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.049+0000 7f6e4c2df640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f6e44148000 con 0x7f6e44148490 2024-04-18T10:06:50.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.049+0000 7f6e4c2df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f6e4406c110 con 0x7f6e440b63b0 2024-04-18T10:06:50.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.049+0000 7f6e4a07b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6e4406bd30 0x7f6e440b82a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42926/0 (socket says 172.21.15.62:42926) 2024-04-18T10:06:50.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.049+0000 7f6e4a07b640 1 -- 172.21.15.62:0/2427964818 learned_addr learned my addr 172.21.15.62:0/2427964818 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:50.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.049+0000 7f6e4a87c640 1 --2- 172.21.15.62:0/2427964818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e440b63b0 0x7f6e440b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4987a640 1 --2- 172.21.15.62:0/2427964818 >> v2:172.21.15.145:3300/0 conn(0x7f6e44148490 0x7f6e44148920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592125831 0 0) 0x7f6e440b6d60 con 0x7f6e4406bd30 2024-04-18T10:06:50.053 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e30003530 con 0x7f6e4406bd30 2024-04-18T10:06:50.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1008630198 0 0) 0x7f6e30003530 con 0x7f6e4406bd30 2024-04-18T10:06:50.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e440b63b0 msgr2=0x7f6e440b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:50.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 --2- 172.21.15.62:0/2427964818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e440b63b0 0x7f6e440b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 >> v2:172.21.15.145:3300/0 conn(0x7f6e44148490 msgr2=0x7f6e44148920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:50.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 --2- 172.21.15.62:0/2427964818 >> v2:172.21.15.145:3300/0 conn(0x7f6e44148490 0x7f6e44148920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e440b8d00 con 0x7f6e4406bd30 2024-04-18T10:06:50.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f6e40001680 con 0x7f6e4406bd30 2024-04-18T10:06:50.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 >> v1:172.21.15.62:6789/0 conn(0x7f6e4406bd30 legacy=0x7f6e440b82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 --2- 172.21.15.62:0/2427964818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e300039c0 0x7f6e30003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 --2- 172.21.15.62:0/2427964818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e30004810 0x7f6e30004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.054 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 --2- 172.21.15.62:0/2427964818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e440b63b0 0x7f6e300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6e44148000 con 0x7f6e30004810 2024-04-18T10:06:50.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4406c110 con 0x7f6e300039c0 2024-04-18T10:06:50.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f6e3000ba50 con 0x7f6e440b63b0 2024-04-18T10:06:50.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4987a640 1 --2- 172.21.15.62:0/2427964818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e30004810 0x7f6e30004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4a07b640 1 --2- 172.21.15.62:0/2427964818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e300039c0 0x7f6e30003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4a87c640 1 --2- 172.21.15.62:0/2427964818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e440b63b0 0x7f6e300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4987a640 1 -- 172.21.15.62:0/2427964818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e440b63b0 msgr2=0x7f6e300051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:50.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4987a640 1 --2- 172.21.15.62:0/2427964818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e440b63b0 0x7f6e300051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4987a640 1 -- 172.21.15.62:0/2427964818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e300039c0 msgr2=0x7f6e30003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:50.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4987a640 1 --2- 172.21.15.62:0/2427964818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e300039c0 0x7f6e30003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.055 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4987a640 1 -- 172.21.15.62:0/2427964818 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e30003530 con 0x7f6e30004810 2024-04-18T10:06:50.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4987a640 1 --2- 172.21.15.62:0/2427964818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e30004810 0x7f6e30004c90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009ae0 tx=0x7f6e3401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f75d65f84df3c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:50.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e340096b0 con 0x7f6e30004810 2024-04-18T10:06:50.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e340a6490 con 0x7f6e30004810 2024-04-18T10:06:50.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e49079640 1 -- 172.21.15.62:0/2427964818 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e340a6880 con 0x7f6e30004810 2024-04-18T10:06:50.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2427964818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e30004810 msgr2=0x7f6e30004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4c2df640 1 --2- 172.21.15.62:0/2427964818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e30004810 0x7f6e30004c90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009ae0 tx=0x7f6e3401cab0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.058 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4a87c640 1 -- 172.21.15.62:0/2427964818 reap_dead start 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2427964818 shutdown_connections 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2427964818 >> 172.21.15.62:0/2427964818 conn(0x7f6e4409c250 msgr2=0x7f6e4409c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2427964818 shutdown_connections 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.053+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2427964818 wait complete. 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4c2df640 1 Processor -- start 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4c2df640 1 -- start start 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4c2df640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e30004810 0x7f6e4414be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4c2df640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e440b23a0 0x7f6e441e0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4c2df640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e441e0c40 0x7f6e4414b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4c2df640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6e44153000 con 0x7f6e440b23a0 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4c2df640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f6e44152e80 con 0x7f6e30004810 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4c2df640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f6e44153180 con 0x7f6e441e0c40 2024-04-18T10:06:50.059 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4987a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e440b23a0 0x7f6e441e0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4987a640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e440b23a0 0x7f6e441e0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56682/0 (socket says 172.21.15.62:56682) 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4987a640 1 -- 172.21.15.62:0/2826688164 learned_addr learned my addr 172.21.15.62:0/2826688164 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4a07b640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e30004810 0x7f6e4414be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4a87c640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e441e0c40 0x7f6e4414b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4987a640 1 -- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e441e0c40 msgr2=0x7f6e4414b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4987a640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e441e0c40 0x7f6e4414b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4987a640 1 -- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e30004810 msgr2=0x7f6e4414be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4987a640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e30004810 0x7f6e4414be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4987a640 1 -- 172.21.15.62:0/2826688164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e440b3790 con 0x7f6e440b23a0 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4987a640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e440b23a0 0x7f6e441e0700 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009ae0 tx=0x7f6e30005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e3affd640 1 -- 172.21.15.62:0/2826688164 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e340be070 con 0x7f6e440b23a0 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e3affd640 1 -- 172.21.15.62:0/2826688164 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e3400f000 con 0x7f6e440b23a0 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2826688164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e440b3d80 con 0x7f6e440b23a0 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e3affd640 1 -- 172.21.15.62:0/2826688164 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e340a7470 con 0x7f6e440b23a0 2024-04-18T10:06:50.060 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2826688164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e440b39c0 con 0x7f6e440b23a0 2024-04-18T10:06:50.061 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2826688164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e44069230 con 0x7f6e440b23a0 2024-04-18T10:06:50.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.057+0000 7f6e3affd640 1 -- 172.21.15.62:0/2826688164 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6e340c1070 con 0x7f6e440b23a0 2024-04-18T10:06:50.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.061+0000 7f6e3affd640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6e28068380 0x7f6e2806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.061+0000 7f6e3affd640 1 -- 172.21.15.62:0/2826688164 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f6e341203d0 con 0x7f6e440b23a0 2024-04-18T10:06:50.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.061+0000 7f6e4a07b640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6e28068380 0x7f6e2806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.062 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.061+0000 7f6e4a07b640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6e28068380 0x7f6e2806a840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6e40005a00 tx=0x7f6e40009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:50.064 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.065+0000 7f6e3affd640 1 -- 172.21.15.62:0/2826688164 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6e340ac020 con 0x7f6e440b23a0 2024-04-18T10:06:50.227 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.225+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2826688164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e440b3f50 con 0x7f6e440b23a0 2024-04-18T10:06:50.228 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e3affd640 1 -- 172.21.15.62:0/2826688164 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16657 (secure 0 0 0) 0x7f6e340167c0 con 0x7f6e440b23a0 2024-04-18T10:06:50.229 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:50.229 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":30,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:39.937405+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:06:36.802689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-18T10:06:37.269237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:50.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6e28068380 msgr2=0x7f6e2806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e4c2df640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6e28068380 0x7f6e2806a840 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6e40005a00 tx=0x7f6e40009040 comp rx=0 tx=0).stop 2024-04-18T10:06:50.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2826688164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e440b23a0 msgr2=0x7f6e441e0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e4c2df640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e440b23a0 0x7f6e441e0700 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6e34009ae0 tx=0x7f6e30005840 comp rx=0 tx=0).stop 2024-04-18T10:06:50.231 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2826688164 shutdown_connections 2024-04-18T10:06:50.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e4c2df640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6e28068380 0x7f6e2806a840 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e4c2df640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e441e0c40 0x7f6e4414b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e4c2df640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6e440b23a0 0x7f6e441e0700 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e4c2df640 1 --2- 172.21.15.62:0/2826688164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e30004810 0x7f6e4414be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2826688164 >> 172.21.15.62:0/2826688164 conn(0x7f6e4409c250 msgr2=0x7f6e440b6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:50.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2826688164 shutdown_connections 2024-04-18T10:06:50.232 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.229+0000 7f6e4c2df640 1 -- 172.21.15.62:0/2826688164 wait complete. 2024-04-18T10:06:50.247 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-18T10:05:47.165426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-18T10:06:36.802689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-04-18T10:06:37.269237+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-18T10:06:50.247 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-18T10:06:50.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.361+0000 7f8aeb5aa640 1 Processor -- start 2024-04-18T10:06:50.363 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.361+0000 7f8aeb5aa640 1 -- start start 2024-04-18T10:06:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8aeb5aa640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8ae406bd30 0x7f8ae40b0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8aeb5aa640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ae406f1d0 con 0x7f8ae406c7b0 2024-04-18T10:06:50.364 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8aeb5aa640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ae40b5de0 con 0x7f8ae40b0ab0 2024-04-18T10:06:50.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8aeb5aa640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f8ae40b6880 con 0x7f8ae406bd30 2024-04-18T10:06:50.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae8b45640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8ae406c7b0 0x7f8ae406e3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42934/0 (socket says 172.21.15.62:42934) 2024-04-18T10:06:50.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae8b45640 1 -- 172.21.15.62:0/632924984 learned_addr learned my addr 172.21.15.62:0/632924984 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:50.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae9346640 1 --2- 172.21.15.62:0/632924984 >> v2:172.21.15.145:3300/0 conn(0x7f8ae406bd30 0x7f8ae40b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3381078937 0 0) 0x7f8ae406f1d0 con 0x7f8ae406c7b0 2024-04-18T10:06:50.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ad0003530 con 0x7f8ae406c7b0 2024-04-18T10:06:50.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1482405206 0 0) 0x7f8ad0003530 con 0x7f8ae406c7b0 2024-04-18T10:06:50.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 >> v2:172.21.15.145:3300/0 conn(0x7f8ae406bd30 msgr2=0x7f8ae40b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:50.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 --2- 172.21.15.62:0/632924984 >> v2:172.21.15.145:3300/0 conn(0x7f8ae406bd30 0x7f8ae40b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 >> v1:172.21.15.145:6789/0 conn(0x7f8ae40b0ab0 legacy=0x7f8ae406eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae40b0f00 con 0x7f8ae406c7b0 2024-04-18T10:06:50.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f8ae0001680 con 0x7f8ae406c7b0 2024-04-18T10:06:50.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 >> v1:172.21.15.62:6789/0 conn(0x7f8ae406c7b0 legacy=0x7f8ae406e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 --2- 172.21.15.62:0/632924984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 0x7f8ad0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 --2- 172.21.15.62:0/632924984 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad0004810 0x7f8ad0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 --2- 172.21.15.62:0/632924984 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad0005220 0x7f8ad000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae40b6880 con 0x7f8ad0003a10 2024-04-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad000c170 con 0x7f8ad0005220 2024-04-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad000c370 con 0x7f8ad0004810 2024-04-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae9346640 1 --2- 172.21.15.62:0/632924984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 0x7f8ad0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae8b45640 1 --2- 172.21.15.62:0/632924984 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad0004810 0x7f8ad0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae9b47640 1 --2- 172.21.15.62:0/632924984 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad0005220 0x7f8ad000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae9346640 1 -- 172.21.15.62:0/632924984 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad0004810 msgr2=0x7f8ad0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae9346640 1 --2- 172.21.15.62:0/632924984 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad0004810 0x7f8ad0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae9346640 1 -- 172.21.15.62:0/632924984 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad0005220 msgr2=0x7f8ad000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae9346640 1 --2- 172.21.15.62:0/632924984 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad0005220 0x7f8ad000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae9346640 1 -- 172.21.15.62:0/632924984 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad0003530 con 0x7f8ad0003a10 2024-04-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae9b47640 1 --2- 172.21.15.62:0/632924984 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad0005220 0x7f8ad000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:50.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.365+0000 7f8ae9b47640 1 -- 172.21.15.62:0/632924984 reap_dead start 2024-04-18T10:06:50.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae9346640 1 --2- 172.21.15.62:0/632924984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 0x7f8ad0003e30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009ab0 tx=0x7f8ad401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0f81842bc3b33e server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:50.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad40096b0 con 0x7f8ad0003a10 2024-04-18T10:06:50.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ad40a6410 con 0x7f8ad0003a10 2024-04-18T10:06:50.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8adbfff640 1 -- 172.21.15.62:0/632924984 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad40a6800 con 0x7f8ad0003a10 2024-04-18T10:06:50.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/632924984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 msgr2=0x7f8ad0003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 --2- 172.21.15.62:0/632924984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 0x7f8ad0003e30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009ab0 tx=0x7f8ad401ca30 comp rx=0 tx=0).stop 2024-04-18T10:06:50.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/632924984 shutdown_connections 2024-04-18T10:06:50.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 --2- 172.21.15.62:0/632924984 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 0x7f8ad0003e30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/632924984 >> 172.21.15.62:0/632924984 conn(0x7f8ae40ba710 msgr2=0x7f8ae40bab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/632924984 shutdown_connections 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/632924984 wait complete. 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 Processor -- start 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 -- start start 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 0x7f8ae41d96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ae41e89c0 0x7f8ae414ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ae41e8e40 0x7f8ae414f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae40afaa0 con 0x7f8ad0003a10 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae40af920 con 0x7f8ae41e8e40 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8ae406ebc0 con 0x7f8ae41e89c0 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae9346640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 0x7f8ae41d96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae9346640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 0x7f8ae41d96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56708/0 (socket says 172.21.15.62:56708) 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae9346640 1 -- 172.21.15.62:0/4138145846 learned_addr learned my addr 172.21.15.62:0/4138145846 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae8b45640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ae41e89c0 0x7f8ae414ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae9b47640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ae41e8e40 0x7f8ae414f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae9346640 1 -- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ae41e89c0 msgr2=0x7f8ae414ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae9346640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ae41e89c0 0x7f8ae414ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae9346640 1 -- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ae41e8e40 msgr2=0x7f8ae414f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae9346640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ae41e8e40 0x7f8ae414f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae9346640 1 -- 172.21.15.62:0/4138145846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae414fbf0 con 0x7f8ad0003a10 2024-04-18T10:06:50.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae9b47640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ae41e8e40 0x7f8ae414f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:50.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ae9346640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 0x7f8ae41d96e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009ab0 tx=0x7f8ad40197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:50.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ad9ffb640 1 -- 172.21.15.62:0/4138145846 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad40b9440 con 0x7f8ad0003a10 2024-04-18T10:06:50.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ad9ffb640 1 -- 172.21.15.62:0/4138145846 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ad40b95e0 con 0x7f8ad0003a10 2024-04-18T10:06:50.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/4138145846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ae4123280 con 0x7f8ad0003a10 2024-04-18T10:06:50.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8ad9ffb640 1 -- 172.21.15.62:0/4138145846 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad40b99b0 con 0x7f8ad0003a10 2024-04-18T10:06:50.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.369+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/4138145846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ae40b2ba0 con 0x7f8ad0003a10 2024-04-18T10:06:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.373+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/4138145846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ae4069230 con 0x7f8ad0003a10 2024-04-18T10:06:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.373+0000 7f8ad9ffb640 1 -- 172.21.15.62:0/4138145846 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8ad40c2440 con 0x7f8ad0003a10 2024-04-18T10:06:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.373+0000 7f8ad9ffb640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8ac8068380 0x7f8ac806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.373+0000 7f8ad9ffb640 1 -- 172.21.15.62:0/4138145846 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f8ad4120bf0 con 0x7f8ad0003a10 2024-04-18T10:06:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.373+0000 7f8ae8b45640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8ac8068380 0x7f8ac806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.373+0000 7f8ae8b45640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8ac8068380 0x7f8ac806a840 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0001ec0 tx=0x7f8ae0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:50.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.377+0000 7f8ad9ffb640 1 -- 172.21.15.62:0/4138145846 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8ad40ac020 con 0x7f8ad0003a10 2024-04-18T10:06:50.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.533+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/4138145846 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8ae414f9c0 con 0x7f8ad0003a10 2024-04-18T10:06:50.533 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.533+0000 7f8ad9ffb640 1 -- 172.21.15.62:0/4138145846 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) ==== 93+0+10 (secure 0 0 0) 0x7f8ad4016740 con 0x7f8ad0003a10 2024-04-18T10:06:50.533 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-04-18T10:06:50.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.533+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8ac8068380 msgr2=0x7f8ac806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.533+0000 7f8aeb5aa640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8ac8068380 0x7f8ac806a840 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0001ec0 tx=0x7f8ae0009040 comp rx=0 tx=0).stop 2024-04-18T10:06:50.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.537+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/4138145846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 msgr2=0x7f8ae41d96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.537+0000 7f8aeb5aa640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 0x7f8ae41d96e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009ab0 tx=0x7f8ad40197d0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.537+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/4138145846 shutdown_connections 2024-04-18T10:06:50.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.537+0000 7f8aeb5aa640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8ac8068380 0x7f8ac806a840 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.537+0000 7f8aeb5aa640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ae41e8e40 0x7f8ae414f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.537+0000 7f8aeb5aa640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ae41e89c0 0x7f8ae414ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.537+0000 7f8aeb5aa640 1 --2- 172.21.15.62:0/4138145846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ad0003a10 0x7f8ae41d96e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.537+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/4138145846 >> 172.21.15.62:0/4138145846 conn(0x7f8ae40ba710 msgr2=0x7f8ae41daa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:50.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.537+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/4138145846 shutdown_connections 2024-04-18T10:06:50.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.537+0000 7f8aeb5aa640 1 -- 172.21.15.62:0/4138145846 wait complete. 2024-04-18T10:06:50.551 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-18T10:06:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdc6a01640 1 Processor -- start 2024-04-18T10:06:50.668 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdc6a01640 1 -- start start 2024-04-18T10:06:50.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdc6a01640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efdc0146690 0x7efdc009e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdc6a01640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdc006d4f0 con 0x7efdc006e490 2024-04-18T10:06:50.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdc6a01640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdc006d7d0 con 0x7efdc00b4c90 2024-04-18T10:06:50.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdc6a01640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7efdc006d2a0 con 0x7efdc0146690 2024-04-18T10:06:50.669 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdb7fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7efdc0146690 0x7efdc009e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdc4f9e640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7efdc006e490 0x7efdc0150410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:42944/0 (socket says 172.21.15.62:42944) 2024-04-18T10:06:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdc4f9e640 1 -- 172.21.15.62:0/28619110 learned_addr learned my addr 172.21.15.62:0/28619110 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdb7fff640 1 -- 172.21.15.62:0/28619110 >> v1:172.21.15.145:6789/0 conn(0x7efdc00b4c90 legacy=0x7efdc0145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdb7fff640 1 -- 172.21.15.62:0/28619110 >> v1:172.21.15.62:6789/0 conn(0x7efdc006e490 legacy=0x7efdc0150410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdbf7fe640 1 -- 172.21.15.62:0/28619110 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3770176223 0 0) 0x7efdc006d4f0 con 0x7efdc006e490 2024-04-18T10:06:50.670 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.669+0000 7efdb7fff640 1 -- 172.21.15.62:0/28619110 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc009c570 con 0x7efdc0146690 2024-04-18T10:06:50.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdb7fff640 1 --2- 172.21.15.62:0/28619110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc0146690 0x7efdc009e090 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efda80097c0 tx=0x7efda801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d67289b49c233c server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:50.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbf7fe640 1 -- 172.21.15.62:0/28619110 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda80b0070 con 0x7efdc0146690 2024-04-18T10:06:50.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbf7fe640 1 -- 172.21.15.62:0/28619110 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efda80ab600 con 0x7efdc0146690 2024-04-18T10:06:50.672 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbf7fe640 1 -- 172.21.15.62:0/28619110 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efda80ab9d0 con 0x7efdc0146690 2024-04-18T10:06:50.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 -- 172.21.15.62:0/28619110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc0146690 msgr2=0x7efdc009e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 --2- 172.21.15.62:0/28619110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc0146690 0x7efdc009e090 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7efda80097c0 tx=0x7efda801c410 comp rx=0 tx=0).stop 2024-04-18T10:06:50.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 -- 172.21.15.62:0/28619110 shutdown_connections 2024-04-18T10:06:50.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 --2- 172.21.15.62:0/28619110 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc0146690 0x7efdc009e090 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 -- 172.21.15.62:0/28619110 >> 172.21.15.62:0/28619110 conn(0x7efdc00b9690 msgr2=0x7efdc00b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:50.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 -- 172.21.15.62:0/28619110 shutdown_connections 2024-04-18T10:06:50.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 -- 172.21.15.62:0/28619110 wait complete. 2024-04-18T10:06:50.673 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 Processor -- start 2024-04-18T10:06:50.674 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 -- start start 2024-04-18T10:06:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc006e490 0x7efdc01e1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdc00b4c90 0x7efdc0140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdc0146690 0x7efdc0141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7efdc01548e0 con 0x7efdc006e490 2024-04-18T10:06:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7efdc0154760 con 0x7efdc0146690 2024-04-18T10:06:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7efdc0154a60 con 0x7efdc00b4c90 2024-04-18T10:06:50.675 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc006e490 0x7efdc01e1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc006e490 0x7efdc01e1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:56720/0 (socket says 172.21.15.62:56720) 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbffff640 1 -- 172.21.15.62:0/68799364 learned_addr learned my addr 172.21.15.62:0/68799364 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbffff640 1 -- 172.21.15.62:0/68799364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdc00b4c90 msgr2=0x7efdc0140b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbffff640 1 --2- 172.21.15.62:0/68799364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdc00b4c90 0x7efdc0140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbffff640 1 -- 172.21.15.62:0/68799364 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdc0146690 msgr2=0x7efdc0141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbffff640 1 --2- 172.21.15.62:0/68799364 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdc0146690 0x7efdc0141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbffff640 1 -- 172.21.15.62:0/68799364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdc01e2560 con 0x7efdc006e490 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbffff640 1 --2- 172.21.15.62:0/68799364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc006e490 0x7efdc01e1ec0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efdb000c530 tx=0x7efdb000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbd7fa640 1 -- 172.21.15.62:0/68799364 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0016020 con 0x7efdc006e490 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbd7fa640 1 -- 172.21.15.62:0/68799364 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdb0005220 con 0x7efdc006e490 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 -- 172.21.15.62:0/68799364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdc01367c0 con 0x7efdc006e490 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdbd7fa640 1 -- 172.21.15.62:0/68799364 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb0007030 con 0x7efdc006e490 2024-04-18T10:06:50.676 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.673+0000 7efdc6a01640 1 -- 172.21.15.62:0/68799364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdc00b3390 con 0x7efdc006e490 2024-04-18T10:06:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.677+0000 7efdc6a01640 1 -- 172.21.15.62:0/68799364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdc0068c00 con 0x7efdc006e490 2024-04-18T10:06:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.677+0000 7efdbd7fa640 1 -- 172.21.15.62:0/68799364 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efdb00071d0 con 0x7efdc006e490 2024-04-18T10:06:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.677+0000 7efdbd7fa640 1 --2- 172.21.15.62:0/68799364 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efda40683b0 0x7efda406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.677+0000 7efdbd7fa640 1 -- 172.21.15.62:0/68799364 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7efdb00862c0 con 0x7efdc006e490 2024-04-18T10:06:50.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.677+0000 7efdb7fff640 1 --2- 172.21.15.62:0/68799364 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efda40683b0 0x7efda406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.677+0000 7efdb7fff640 1 --2- 172.21.15.62:0/68799364 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efda40683b0 0x7efda406a870 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efda80ad000 tx=0x7efda8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:50.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.681+0000 7efdbd7fa640 1 -- 172.21.15.62:0/68799364 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efdb0020020 con 0x7efdc006e490 2024-04-18T10:06:50.837 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.837+0000 7efdc6a01640 1 -- 172.21.15.62:0/68799364 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efdc00b44c0 con 0x7efdc006e490 2024-04-18T10:06:50.838 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.837+0000 7efdbd7fa640 1 -- 172.21.15.62:0/68799364 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v30) ==== 104+0+11 (secure 0 0 0) 0x7efdb00074c0 con 0x7efdc006e490 2024-04-18T10:06:50.838 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 32 2024-04-18T10:06:50.840 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.841+0000 7efdc6a01640 1 -- 172.21.15.62:0/68799364 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efda40683b0 msgr2=0x7efda406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.841+0000 7efdc6a01640 1 --2- 172.21.15.62:0/68799364 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efda40683b0 0x7efda406a870 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efda80ad000 tx=0x7efda8008040 comp rx=0 tx=0).stop 2024-04-18T10:06:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.841+0000 7efdc6a01640 1 -- 172.21.15.62:0/68799364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc006e490 msgr2=0x7efdc01e1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.841+0000 7efdc6a01640 1 --2- 172.21.15.62:0/68799364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc006e490 0x7efdc01e1ec0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7efdb000c530 tx=0x7efdb000c9f0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.841+0000 7efdc6a01640 1 -- 172.21.15.62:0/68799364 shutdown_connections 2024-04-18T10:06:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.841+0000 7efdc6a01640 1 --2- 172.21.15.62:0/68799364 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7efda40683b0 0x7efda406a870 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.841+0000 7efdc6a01640 1 --2- 172.21.15.62:0/68799364 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efdc0146690 0x7efdc0141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.841+0000 7efdc6a01640 1 --2- 172.21.15.62:0/68799364 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efdc00b4c90 0x7efdc0140b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.841+0000 7efdc6a01640 1 --2- 172.21.15.62:0/68799364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efdc006e490 0x7efdc01e1ec0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.841+0000 7efdc6a01640 1 -- 172.21.15.62:0/68799364 >> 172.21.15.62:0/68799364 conn(0x7efdc00b9690 msgr2=0x7efdc009cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.841+0000 7efdc6a01640 1 -- 172.21.15.62:0/68799364 shutdown_connections 2024-04-18T10:06:50.841 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.841+0000 7efdc6a01640 1 -- 172.21.15.62:0/68799364 wait complete. 2024-04-18T10:06:50.856 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-18T10:06:50.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.969+0000 7f59d1c0b640 1 Processor -- start 2024-04-18T10:06:50.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.969+0000 7f59d1c0b640 1 -- start start 2024-04-18T10:06:50.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59d1c0b640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f59cc146690 0x7f59cc09e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59d1c0b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59cc06d4f0 con 0x7f59cc06e490 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59d1c0b640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59cc06d7d0 con 0x7f59cc0b4c90 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59d1c0b640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f59cc06d2a0 con 0x7f59cc146690 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59cbfff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f59cc06e490 0x7f59cc150410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48664/0 (socket says 172.21.15.62:48664) 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59cbfff640 1 -- 172.21.15.62:0/2129724046 learned_addr learned my addr 172.21.15.62:0/2129724046 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59caffd640 1 --2- 172.21.15.62:0/2129724046 >> v2:172.21.15.145:3300/0 conn(0x7f59cc146690 0x7f59cc09e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2490676830 0 0) 0x7f59cc06d4f0 con 0x7f59cc06e490 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59b4003530 con 0x7f59cc06e490 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3050219268 0 0) 0x7f59b4003530 con 0x7f59cc06e490 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 >> v2:172.21.15.145:3300/0 conn(0x7f59cc146690 msgr2=0x7f59cc09e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 --2- 172.21.15.62:0/2129724046 >> v2:172.21.15.145:3300/0 conn(0x7f59cc146690 0x7f59cc09e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 >> v1:172.21.15.145:6789/0 conn(0x7f59cc0b4c90 legacy=0x7f59cc145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59cc09c570 con 0x7f59cc06e490 2024-04-18T10:06:50.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f59c4001680 con 0x7f59cc06e490 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 >> v1:172.21.15.62:6789/0 conn(0x7f59cc06e490 legacy=0x7f59cc150410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 --2- 172.21.15.62:0/2129724046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59b40039c0 0x7f59b4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 --2- 172.21.15.62:0/2129724046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59b4004810 0x7f59b4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 --2- 172.21.15.62:0/2129724046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59b40051d0 0x7f59b400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc06d2a0 con 0x7f59b40039c0 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f59b400c170 con 0x7f59b40051d0 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f59b400c370 con 0x7f59b4004810 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59cb7fe640 1 --2- 172.21.15.62:0/2129724046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59b40039c0 0x7f59b4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59caffd640 1 --2- 172.21.15.62:0/2129724046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59b4004810 0x7f59b4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59cbfff640 1 --2- 172.21.15.62:0/2129724046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59b40051d0 0x7f59b400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59cb7fe640 1 -- 172.21.15.62:0/2129724046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59b4004810 msgr2=0x7f59b4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59cb7fe640 1 --2- 172.21.15.62:0/2129724046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59b4004810 0x7f59b4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59cb7fe640 1 -- 172.21.15.62:0/2129724046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59b40051d0 msgr2=0x7f59b400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:50.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59cb7fe640 1 --2- 172.21.15.62:0/2129724046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59b40051d0 0x7f59b400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59cb7fe640 1 -- 172.21.15.62:0/2129724046 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b4003530 con 0x7f59b40039c0 2024-04-18T10:06:50.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59cbfff640 1 --2- 172.21.15.62:0/2129724046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59b40051d0 0x7f59b400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:50.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.973+0000 7f59cbfff640 1 -- 172.21.15.62:0/2129724046 reap_dead start 2024-04-18T10:06:50.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59cb7fe640 1 --2- 172.21.15.62:0/2129724046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59b40039c0 0x7f59b4003de0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f59bc001e70 tx=0x7f59bc01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54bc6cc26a20553d server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:50.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc001a90 con 0x7f59b40039c0 2024-04-18T10:06:50.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59bc003b80 con 0x7f59b40039c0 2024-04-18T10:06:50.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59ca7fc640 1 -- 172.21.15.62:0/2129724046 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59bc0050d0 con 0x7f59b40039c0 2024-04-18T10:06:50.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 -- 172.21.15.62:0/2129724046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59b40039c0 msgr2=0x7f59b4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 --2- 172.21.15.62:0/2129724046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59b40039c0 0x7f59b4003de0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f59bc001e70 tx=0x7f59bc01c760 comp rx=0 tx=0).stop 2024-04-18T10:06:50.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 -- 172.21.15.62:0/2129724046 shutdown_connections 2024-04-18T10:06:50.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 --2- 172.21.15.62:0/2129724046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59b40039c0 0x7f59b4003de0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 -- 172.21.15.62:0/2129724046 >> 172.21.15.62:0/2129724046 conn(0x7f59cc0b9690 msgr2=0x7f59cc0b9aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:50.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 -- 172.21.15.62:0/2129724046 shutdown_connections 2024-04-18T10:06:50.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 -- 172.21.15.62:0/2129724046 wait complete. 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 Processor -- start 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 -- start start 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59b40039c0 0x7f59cc1e45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59cc1d4930 0x7f59cc0b89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59cc13e990 0x7f59cc13ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc149220 con 0x7f59cc13e990 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f59cc1490a0 con 0x7f59b40039c0 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f59cc1493a0 con 0x7f59cc1d4930 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59cc13e990 0x7f59cc13ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59cbfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59cc13e990 0x7f59cc13ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48810/0 (socket says 172.21.15.62:48810) 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59cbfff640 1 -- 172.21.15.62:0/1497920953 learned_addr learned my addr 172.21.15.62:0/1497920953 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59cb7fe640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59b40039c0 0x7f59cc1e45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59caffd640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59cc1d4930 0x7f59cc0b89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59cbfff640 1 -- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59cc1d4930 msgr2=0x7f59cc0b89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59cbfff640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59cc1d4930 0x7f59cc0b89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59cbfff640 1 -- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59b40039c0 msgr2=0x7f59cc1e45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59cbfff640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59b40039c0 0x7f59cc1e45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59cbfff640 1 -- 172.21.15.62:0/1497920953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59cc0af7c0 con 0x7f59cc13e990 2024-04-18T10:06:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59cbfff640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59cc13e990 0x7f59cc13ee10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f59cc09ca80 tx=0x7f59c400d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59b3fff640 1 -- 172.21.15.62:0/1497920953 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c4007440 con 0x7f59cc13e990 2024-04-18T10:06:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59b3fff640 1 -- 172.21.15.62:0/1497920953 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59c40075e0 con 0x7f59cc13e990 2024-04-18T10:06:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59b3fff640 1 -- 172.21.15.62:0/1497920953 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c40049f0 con 0x7f59cc13e990 2024-04-18T10:06:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 -- 172.21.15.62:0/1497920953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59cc0b81a0 con 0x7f59cc13e990 2024-04-18T10:06:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.977+0000 7f59d1c0b640 1 -- 172.21.15.62:0/1497920953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59cc13f350 con 0x7f59cc13e990 2024-04-18T10:06:50.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.981+0000 7f59d1c0b640 1 -- 172.21.15.62:0/1497920953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59cc068c00 con 0x7f59cc13e990 2024-04-18T10:06:50.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.981+0000 7f59b3fff640 1 -- 172.21.15.62:0/1497920953 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f59c4004b90 con 0x7f59cc13e990 2024-04-18T10:06:50.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.981+0000 7f59b3fff640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f59b8068380 0x7f59b806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:50.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.981+0000 7f59b3fff640 1 -- 172.21.15.62:0/1497920953 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f59c4086490 con 0x7f59cc13e990 2024-04-18T10:06:50.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.981+0000 7f59cb7fe640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f59b8068380 0x7f59b806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:50.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.981+0000 7f59cb7fe640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f59b8068380 0x7f59b806a840 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f59bc0ac000 tx=0x7f59bc001520 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:50.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:50.985+0000 7f59b3fff640 1 -- 172.21.15.62:0/1497920953 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f59c401a060 con 0x7f59cc13e990 2024-04-18T10:06:51.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.141+0000 7f59d1c0b640 1 -- 172.21.15.62:0/1497920953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f59cc0b44c0 con 0x7f59cc13e990 2024-04-18T10:06:51.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.141+0000 7f59b3fff640 1 -- 172.21.15.62:0/1497920953 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v30) ==== 100+0+11 (secure 0 0 0) 0x7f59c4056ae0 con 0x7f59cc13e990 2024-04-18T10:06:51.143 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 64 2024-04-18T10:06:51.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.145+0000 7f59d1c0b640 1 -- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f59b8068380 msgr2=0x7f59b806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.145+0000 7f59d1c0b640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f59b8068380 0x7f59b806a840 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f59bc0ac000 tx=0x7f59bc001520 comp rx=0 tx=0).stop 2024-04-18T10:06:51.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.145+0000 7f59d1c0b640 1 -- 172.21.15.62:0/1497920953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59cc13e990 msgr2=0x7f59cc13ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.145+0000 7f59d1c0b640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59cc13e990 0x7f59cc13ee10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f59cc09ca80 tx=0x7f59c400d950 comp rx=0 tx=0).stop 2024-04-18T10:06:51.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.145+0000 7f59d1c0b640 1 -- 172.21.15.62:0/1497920953 shutdown_connections 2024-04-18T10:06:51.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.145+0000 7f59d1c0b640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f59b8068380 0x7f59b806a840 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.145+0000 7f59d1c0b640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59cc13e990 0x7f59cc13ee10 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.145+0000 7f59d1c0b640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59cc1d4930 0x7f59cc0b89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.145+0000 7f59d1c0b640 1 --2- 172.21.15.62:0/1497920953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59b40039c0 0x7f59cc1e45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.145+0000 7f59d1c0b640 1 -- 172.21.15.62:0/1497920953 >> 172.21.15.62:0/1497920953 conn(0x7f59cc0b9690 msgr2=0x7f59cc0b6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:51.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.145+0000 7f59d1c0b640 1 -- 172.21.15.62:0/1497920953 shutdown_connections 2024-04-18T10:06:51.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.145+0000 7f59d1c0b640 1 -- 172.21.15.62:0/1497920953 wait complete. 2024-04-18T10:06:51.160 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:51.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.273+0000 7f63f319c640 1 Processor -- start 2024-04-18T10:06:51.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.273+0000 7f63f319c640 1 -- start start 2024-04-18T10:06:51.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.273+0000 7f63f319c640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f63ec13ffc0 0x7f63ec149150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.273+0000 7f63f319c640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63ec0b8750 con 0x7f63ec09ce60 2024-04-18T10:06:51.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.273+0000 7f63f319c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63ec0b89f0 con 0x7f63ec13fbe0 2024-04-18T10:06:51.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.273+0000 7f63f319c640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f63ec0b8cd0 con 0x7f63ec13ffc0 2024-04-18T10:06:51.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63f1739640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f63ec09ce60 0x7f63ec1503e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48674/0 (socket says 172.21.15.62:48674) 2024-04-18T10:06:51.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63f1739640 1 -- 172.21.15.62:0/1701288417 learned_addr learned my addr 172.21.15.62:0/1701288417 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:51.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2107484642 0 0) 0x7f63ec0b8750 con 0x7f63ec09ce60 2024-04-18T10:06:51.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63ebfff640 1 --2- 172.21.15.62:0/1701288417 >> v2:172.21.15.145:3300/0 conn(0x7f63ec13ffc0 0x7f63ec149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63dc003530 con 0x7f63ec09ce60 2024-04-18T10:06:51.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1405720061 0 0) 0x7f63dc003530 con 0x7f63ec09ce60 2024-04-18T10:06:51.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 >> v2:172.21.15.145:3300/0 conn(0x7f63ec13ffc0 msgr2=0x7f63ec149150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:51.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 --2- 172.21.15.62:0/1701288417 >> v2:172.21.15.145:3300/0 conn(0x7f63ec13ffc0 0x7f63ec149150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 >> v1:172.21.15.145:6789/0 conn(0x7f63ec13fbe0 legacy=0x7f63ec06bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ec06fa90 con 0x7f63ec09ce60 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f63e4001a00 con 0x7f63ec09ce60 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 >> v1:172.21.15.62:6789/0 conn(0x7f63ec09ce60 legacy=0x7f63ec1503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 --2- 172.21.15.62:0/1701288417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63dc003a10 0x7f63dc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 --2- 172.21.15.62:0/1701288417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63dc004810 0x7f63dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 --2- 172.21.15.62:0/1701288417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63dc005220 0x7f63dc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec0b8cd0 con 0x7f63dc005220 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc00c170 con 0x7f63dc004810 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f63dc00c370 con 0x7f63dc003a10 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63f1739640 1 --2- 172.21.15.62:0/1701288417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63dc005220 0x7f63dc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63ebfff640 1 --2- 172.21.15.62:0/1701288417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63dc004810 0x7f63dc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63f0f38640 1 --2- 172.21.15.62:0/1701288417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63dc003a10 0x7f63dc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63f1739640 1 -- 172.21.15.62:0/1701288417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63dc003a10 msgr2=0x7f63dc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:51.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63f1739640 1 --2- 172.21.15.62:0/1701288417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63dc003a10 0x7f63dc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63f1739640 1 -- 172.21.15.62:0/1701288417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63dc004810 msgr2=0x7f63dc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:51.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63f1739640 1 --2- 172.21.15.62:0/1701288417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63dc004810 0x7f63dc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63f1739640 1 -- 172.21.15.62:0/1701288417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc003530 con 0x7f63dc005220 2024-04-18T10:06:51.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63f1739640 1 -- 172.21.15.62:0/1701288417 reap_dead start 2024-04-18T10:06:51.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63f1739640 1 --2- 172.21.15.62:0/1701288417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63dc005220 0x7f63dc00b9d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f63e400a780 tx=0x7f63e401fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607a4b5077e56054 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:51.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e400a3d0 con 0x7f63dc005220 2024-04-18T10:06:51.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63e40aa430 con 0x7f63dc005220 2024-04-18T10:06:51.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.277+0000 7f63eb7fe640 1 -- 172.21.15.62:0/1701288417 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e40aa800 con 0x7f63dc005220 2024-04-18T10:06:51.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 -- 172.21.15.62:0/1701288417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63dc005220 msgr2=0x7f63dc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 --2- 172.21.15.62:0/1701288417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63dc005220 0x7f63dc00b9d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f63e400a780 tx=0x7f63e401fa00 comp rx=0 tx=0).stop 2024-04-18T10:06:51.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 -- 172.21.15.62:0/1701288417 shutdown_connections 2024-04-18T10:06:51.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 --2- 172.21.15.62:0/1701288417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63dc005220 0x7f63dc00b9d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 -- 172.21.15.62:0/1701288417 >> 172.21.15.62:0/1701288417 conn(0x7f63ec09c250 msgr2=0x7f63ec09c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:51.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 -- 172.21.15.62:0/1701288417 shutdown_connections 2024-04-18T10:06:51.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 -- 172.21.15.62:0/1701288417 wait complete. 2024-04-18T10:06:51.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 Processor -- start 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 -- start start 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63dc005220 0x7f63ec1e0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63ec09faa0 0x7f63ec09e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63ec09ea10 0x7f63ec09ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec06bad0 con 0x7f63ec09faa0 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec06f8c0 con 0x7f63ec09ea10 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f63ec06bc50 con 0x7f63dc005220 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63ebfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63ec09faa0 0x7f63ec09e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63ebfff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63ec09faa0 0x7f63ec09e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48824/0 (socket says 172.21.15.62:48824) 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63ebfff640 1 -- 172.21.15.62:0/2019345687 learned_addr learned my addr 172.21.15.62:0/2019345687 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f0f38640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63dc005220 0x7f63ec1e0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63ebfff640 1 -- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63dc005220 msgr2=0x7f63ec1e0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63ebfff640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63dc005220 0x7f63ec1e0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63ebfff640 1 -- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63ec09ea10 msgr2=0x7f63ec09ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63ebfff640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63ec09ea10 0x7f63ec09ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63ebfff640 1 -- 172.21.15.62:0/2019345687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ec1e1070 con 0x7f63ec09faa0 2024-04-18T10:06:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f0f38640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63dc005220 0x7f63ec1e0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63ebfff640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63ec09faa0 0x7f63ec09e4d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f63e0011990 tx=0x7f63e0011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63e97fa640 1 -- 172.21.15.62:0/2019345687 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e000fa70 con 0x7f63ec09faa0 2024-04-18T10:06:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63e97fa640 1 -- 172.21.15.62:0/2019345687 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63e000fc10 con 0x7f63ec09faa0 2024-04-18T10:06:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63e97fa640 1 -- 172.21.15.62:0/2019345687 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e0014440 con 0x7f63ec09faa0 2024-04-18T10:06:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 -- 172.21.15.62:0/2019345687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ec073fb0 con 0x7f63ec09faa0 2024-04-18T10:06:51.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 -- 172.21.15.62:0/2019345687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ec1465e0 con 0x7f63ec09faa0 2024-04-18T10:06:51.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.281+0000 7f63f319c640 1 -- 172.21.15.62:0/2019345687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63ec0b5a90 con 0x7f63ec09faa0 2024-04-18T10:06:51.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.285+0000 7f63e97fa640 1 -- 172.21.15.62:0/2019345687 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f63e0012040 con 0x7f63ec09faa0 2024-04-18T10:06:51.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.285+0000 7f63e97fa640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f63d0068380 0x7f63d006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.285+0000 7f63e97fa640 1 -- 172.21.15.62:0/2019345687 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f63e0086f00 con 0x7f63ec09faa0 2024-04-18T10:06:51.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.285+0000 7f63f0f38640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f63d0068380 0x7f63d006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.285 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.285+0000 7f63f0f38640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f63d0068380 0x7f63d006a840 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f63d8007fd0 tx=0x7f63d8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:51.288 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.289+0000 7f63e97fa640 1 -- 172.21.15.62:0/2019345687 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f63ec1e1070 con 0x7f63ec09faa0 2024-04-18T10:06:51.488 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.489+0000 7f63f319c640 1 -- 172.21.15.62:0/2019345687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63ec145120 con 0x7f63ec09faa0 2024-04-18T10:06:51.489 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.489+0000 7f63e97fa640 1 -- 172.21.15.62:0/2019345687 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+3534 (secure 0 0 0) 0x7f63e0019030 con 0x7f63ec09faa0 2024-04-18T10:06:51.490 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:51.490 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4705,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1649467687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1649467687},{"type":"v1","addr":"172.21.15.145:6835","nonce":1649467687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:06:38.304481+0000","modified":"2024-04-18T10:06:41.339980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4932":{"gid":4932,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2971073685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2971073685},{"type":"v1","addr":"172.21.15.62:6833","nonce":2971073685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4932,"qdb_cluster":[4932]},"id":2}]} 2024-04-18T10:06:51.490 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-04-18T10:06:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.489+0000 7f63f319c640 1 -- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f63d0068380 msgr2=0x7f63d006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.489+0000 7f63f319c640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f63d0068380 0x7f63d006a840 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f63d8007fd0 tx=0x7f63d8009040 comp rx=0 tx=0).stop 2024-04-18T10:06:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.489+0000 7f63f319c640 1 -- 172.21.15.62:0/2019345687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63ec09faa0 msgr2=0x7f63ec09e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.489+0000 7f63f319c640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63ec09faa0 0x7f63ec09e4d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f63e0011990 tx=0x7f63e0011e50 comp rx=0 tx=0).stop 2024-04-18T10:06:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.489+0000 7f63f319c640 1 -- 172.21.15.62:0/2019345687 shutdown_connections 2024-04-18T10:06:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.489+0000 7f63f319c640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f63d0068380 0x7f63d006a840 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.492 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.489+0000 7f63f319c640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63ec09ea10 0x7f63ec09ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.489+0000 7f63f319c640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63ec09faa0 0x7f63ec09e4d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.489+0000 7f63f319c640 1 --2- 172.21.15.62:0/2019345687 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63dc005220 0x7f63ec1e0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.489+0000 7f63f319c640 1 -- 172.21.15.62:0/2019345687 >> 172.21.15.62:0/2019345687 conn(0x7f63ec09c250 msgr2=0x7f63ec09a130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.493+0000 7f63f319c640 1 -- 172.21.15.62:0/2019345687 shutdown_connections 2024-04-18T10:06:51.493 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.493+0000 7f63f319c640 1 -- 172.21.15.62:0/2019345687 wait complete. 2024-04-18T10:06:51.508 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.621+0000 7f1d5e69f640 1 Processor -- start 2024-04-18T10:06:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.621+0000 7f1d5e69f640 1 -- start start 2024-04-18T10:06:51.623 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.621+0000 7f1d5e69f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1d580b4970 0x7f1d581442c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.621+0000 7f1d5e69f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d580ba170 con 0x7f1d580b44e0 2024-04-18T10:06:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.621+0000 7f1d5e69f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d581458a0 con 0x7f1d580b9d90 2024-04-18T10:06:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.621+0000 7f1d5e69f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f1d580b4de0 con 0x7f1d580b4970 2024-04-18T10:06:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d5cc3c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1d580b4970 0x7f1d581442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d577fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1d580b44e0 0x7f1d58143bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48676/0 (socket says 172.21.15.62:48676) 2024-04-18T10:06:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d577fe640 1 -- 172.21.15.62:0/1678482076 learned_addr learned my addr 172.21.15.62:0/1678482076 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d5cc3c640 1 -- 172.21.15.62:0/1678482076 >> v1:172.21.15.145:6789/0 conn(0x7f1d580b9d90 legacy=0x7f1d581450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d5cc3c640 1 -- 172.21.15.62:0/1678482076 >> v1:172.21.15.62:6789/0 conn(0x7f1d580b44e0 legacy=0x7f1d58143bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.624 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d56ffd640 1 -- 172.21.15.62:0/1678482076 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 721328114 0 0) 0x7f1d580ba170 con 0x7f1d580b44e0 2024-04-18T10:06:51.625 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d5cc3c640 1 -- 172.21.15.62:0/1678482076 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d580b3fc0 con 0x7f1d580b4970 2024-04-18T10:06:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d5cc3c640 1 --2- 172.21.15.62:0/1678482076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d580b4970 0x7f1d581442c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1d500097c0 tx=0x7f1d5001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4fcc04e478dd52 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d56ffd640 1 -- 172.21.15.62:0/1678482076 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d500b0070 con 0x7f1d580b4970 2024-04-18T10:06:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d56ffd640 1 -- 172.21.15.62:0/1678482076 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d500ab600 con 0x7f1d580b4970 2024-04-18T10:06:51.626 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d56ffd640 1 -- 172.21.15.62:0/1678482076 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d500ab9d0 con 0x7f1d580b4970 2024-04-18T10:06:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d5e69f640 1 -- 172.21.15.62:0/1678482076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d580b4970 msgr2=0x7f1d581442c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d5e69f640 1 --2- 172.21.15.62:0/1678482076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d580b4970 0x7f1d581442c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1d500097c0 tx=0x7f1d5001c410 comp rx=0 tx=0).stop 2024-04-18T10:06:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d5e69f640 1 -- 172.21.15.62:0/1678482076 shutdown_connections 2024-04-18T10:06:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d5e69f640 1 --2- 172.21.15.62:0/1678482076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d580b4970 0x7f1d581442c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d5e69f640 1 -- 172.21.15.62:0/1678482076 >> 172.21.15.62:0/1678482076 conn(0x7f1d5806c0d0 msgr2=0x7f1d580b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:51.627 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d5e69f640 1 -- 172.21.15.62:0/1678482076 shutdown_connections 2024-04-18T10:06:51.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.625+0000 7f1d5e69f640 1 -- 172.21.15.62:0/1678482076 wait complete. 2024-04-18T10:06:51.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d5e69f640 1 Processor -- start 2024-04-18T10:06:51.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d5e69f640 1 -- start start 2024-04-18T10:06:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d5e69f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d580b44e0 0x7f1d5813eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d5e69f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d580b4970 0x7f1d5813c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d5e69f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d580b9d90 0x7f1d581e4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d5e69f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5806d620 con 0x7f1d580b44e0 2024-04-18T10:06:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d5e69f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f1d5806d4a0 con 0x7f1d580b4970 2024-04-18T10:06:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d5e69f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f1d5806d7a0 con 0x7f1d580b9d90 2024-04-18T10:06:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d57fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d580b44e0 0x7f1d5813eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d57fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d580b44e0 0x7f1d5813eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48834/0 (socket says 172.21.15.62:48834) 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d57fff640 1 -- 172.21.15.62:0/694405180 learned_addr learned my addr 172.21.15.62:0/694405180 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d577fe640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d580b4970 0x7f1d5813c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d5cc3c640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d580b9d90 0x7f1d581e4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d57fff640 1 -- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d580b9d90 msgr2=0x7f1d581e4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d57fff640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d580b9d90 0x7f1d581e4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d57fff640 1 -- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d580b4970 msgr2=0x7f1d5813c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d57fff640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d580b4970 0x7f1d5813c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d57fff640 1 -- 172.21.15.62:0/694405180 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d5813cf90 con 0x7f1d580b44e0 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d57fff640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d580b44e0 0x7f1d5813eb90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800c3f0 tx=0x7f1d4800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d54ff9640 1 -- 172.21.15.62:0/694405180 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d48017070 con 0x7f1d580b44e0 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d54ff9640 1 -- 172.21.15.62:0/694405180 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d480052d0 con 0x7f1d580b44e0 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d54ff9640 1 -- 172.21.15.62:0/694405180 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d48003840 con 0x7f1d580b44e0 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d5e69f640 1 -- 172.21.15.62:0/694405180 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d581e3b00 con 0x7f1d580b44e0 2024-04-18T10:06:51.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d5e69f640 1 -- 172.21.15.62:0/694405180 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d5813d530 con 0x7f1d580b44e0 2024-04-18T10:06:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d5e69f640 1 -- 172.21.15.62:0/694405180 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d5813cda0 con 0x7f1d580b44e0 2024-04-18T10:06:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.629+0000 7f1d54ff9640 1 -- 172.21.15.62:0/694405180 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1d4801a070 con 0x7f1d580b44e0 2024-04-18T10:06:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.633+0000 7f1d54ff9640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1d44068380 0x7f1d4406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.633+0000 7f1d54ff9640 1 -- 172.21.15.62:0/694405180 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f1d48085c90 con 0x7f1d580b44e0 2024-04-18T10:06:51.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.633+0000 7f1d577fe640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1d44068380 0x7f1d4406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.633+0000 7f1d577fe640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1d44068380 0x7f1d4406a840 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c007fd0 tx=0x7f1d4c007f20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:51.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.637+0000 7f1d54ff9640 1 -- 172.21.15.62:0/694405180 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1d48005480 con 0x7f1d580b44e0 2024-04-18T10:06:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.793+0000 7f1d5e69f640 1 -- 172.21.15.62:0/694405180 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1d58068b90 con 0x7f1d580b44e0 2024-04-18T10:06:51.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d54ff9640 1 -- 172.21.15.62:0/694405180 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16657 (secure 0 0 0) 0x7f1d480562e0 con 0x7f1d580b44e0 2024-04-18T10:06:51.797 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:51.797 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":30,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:39.937405+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:06:36.802689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-18T10:06:37.269237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:51.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d5e69f640 1 -- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1d44068380 msgr2=0x7f1d4406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d5e69f640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1d44068380 0x7f1d4406a840 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c007fd0 tx=0x7f1d4c007f20 comp rx=0 tx=0).stop 2024-04-18T10:06:51.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d5e69f640 1 -- 172.21.15.62:0/694405180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d580b44e0 msgr2=0x7f1d5813eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d5e69f640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d580b44e0 0x7f1d5813eb90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1d4800c3f0 tx=0x7f1d4800c8b0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d5e69f640 1 -- 172.21.15.62:0/694405180 shutdown_connections 2024-04-18T10:06:51.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d5e69f640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1d44068380 0x7f1d4406a840 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d5e69f640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d580b9d90 0x7f1d581e4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d5e69f640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d580b4970 0x7f1d5813c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d5e69f640 1 --2- 172.21.15.62:0/694405180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d580b44e0 0x7f1d5813eb90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d5e69f640 1 -- 172.21.15.62:0/694405180 >> 172.21.15.62:0/694405180 conn(0x7f1d5806c0d0 msgr2=0x7f1d580b7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:51.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d5e69f640 1 -- 172.21.15.62:0/694405180 shutdown_connections 2024-04-18T10:06:51.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.797+0000 7f1d5e69f640 1 -- 172.21.15.62:0/694405180 wait complete. 2024-04-18T10:06:51.815 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-18T10:06:51.815 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:51.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.933+0000 7f3f570bd640 1 Processor -- start 2024-04-18T10:06:51.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.933+0000 7f3f570bd640 1 -- start start 2024-04-18T10:06:51.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f570bd640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3f500b4970 0x7f3f501442c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f570bd640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f500ba170 con 0x7f3f500b44e0 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f570bd640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f501458a0 con 0x7f3f500b9d90 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f570bd640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f3f500b4de0 con 0x7f3f500b4970 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4ffff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f3f500b44e0 0x7f3f50143bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48692/0 (socket says 172.21.15.62:48692) 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4ffff640 1 -- 172.21.15.62:0/1997365058 learned_addr learned my addr 172.21.15.62:0/1997365058 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f5565a640 1 --2- 172.21.15.62:0/1997365058 >> v2:172.21.15.145:3300/0 conn(0x7f3f500b4970 0x7f3f501442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4030379114 0 0) 0x7f3f500ba170 con 0x7f3f500b44e0 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f3c003530 con 0x7f3f500b44e0 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1997056927 0 0) 0x7f3f3c003530 con 0x7f3f500b44e0 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 >> v2:172.21.15.145:3300/0 conn(0x7f3f500b4970 msgr2=0x7f3f501442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 --2- 172.21.15.62:0/1997365058 >> v2:172.21.15.145:3300/0 conn(0x7f3f500b4970 0x7f3f501442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 >> v1:172.21.15.145:6789/0 conn(0x7f3f500b9d90 legacy=0x7f3f501450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f500b3fc0 con 0x7f3f500b44e0 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f3f38001680 con 0x7f3f500b44e0 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 >> v1:172.21.15.62:6789/0 conn(0x7f3f500b44e0 legacy=0x7f3f50143bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 --2- 172.21.15.62:0/1997365058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f3c0039c0 0x7f3f3c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 --2- 172.21.15.62:0/1997365058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f3c004810 0x7f3f3c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 --2- 172.21.15.62:0/1997365058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f3c0051d0 0x7f3f3c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3f500b4de0 con 0x7f3f3c0051d0 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3c00c170 con 0x7f3f3c0039c0 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3f3c00c370 con 0x7f3f3c004810 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f5565a640 1 --2- 172.21.15.62:0/1997365058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f3c0051d0 0x7f3f3c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4ffff640 1 --2- 172.21.15.62:0/1997365058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f3c004810 0x7f3f3c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f54e59640 1 --2- 172.21.15.62:0/1997365058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f3c0039c0 0x7f3f3c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f5565a640 1 -- 172.21.15.62:0/1997365058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f3c004810 msgr2=0x7f3f3c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f5565a640 1 --2- 172.21.15.62:0/1997365058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f3c004810 0x7f3f3c004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f5565a640 1 -- 172.21.15.62:0/1997365058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f3c0039c0 msgr2=0x7f3f3c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:51.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f5565a640 1 --2- 172.21.15.62:0/1997365058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f3c0039c0 0x7f3f3c003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f5565a640 1 -- 172.21.15.62:0/1997365058 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3c003530 con 0x7f3f3c0051d0 2024-04-18T10:06:51.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f5565a640 1 -- 172.21.15.62:0/1997365058 reap_dead start 2024-04-18T10:06:51.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f5565a640 1 --2- 172.21.15.62:0/1997365058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f3c0051d0 0x7f3f3c00b9d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009fa0 tx=0x7f3f4801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=703e4b12d26e27df server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:51.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f480b2070 con 0x7f3f3c0051d0 2024-04-18T10:06:51.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f480a5d50 con 0x7f3f3c0051d0 2024-04-18T10:06:51.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.937+0000 7f3f4f7fe640 1 -- 172.21.15.62:0/1997365058 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48009bf0 con 0x7f3f3c0051d0 2024-04-18T10:06:51.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 -- 172.21.15.62:0/1997365058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f3c0051d0 msgr2=0x7f3f3c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 --2- 172.21.15.62:0/1997365058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f3c0051d0 0x7f3f3c00b9d0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009fa0 tx=0x7f3f4801cc80 comp rx=0 tx=0).stop 2024-04-18T10:06:51.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 -- 172.21.15.62:0/1997365058 shutdown_connections 2024-04-18T10:06:51.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 --2- 172.21.15.62:0/1997365058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f3c0051d0 0x7f3f3c00b9d0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 -- 172.21.15.62:0/1997365058 >> 172.21.15.62:0/1997365058 conn(0x7f3f5006c0d0 msgr2=0x7f3f500b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:51.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 -- 172.21.15.62:0/1997365058 shutdown_connections 2024-04-18T10:06:51.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 -- 172.21.15.62:0/1997365058 wait complete. 2024-04-18T10:06:51.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 Processor -- start 2024-04-18T10:06:51.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 -- start start 2024-04-18T10:06:51.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f3c0051d0 0x7f3f50091800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f501e0770 0x7f3f501e0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f501dd1c0 0x7f3f501dd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5009e3d0 con 0x7f3f501e0770 2024-04-18T10:06:51.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f3f50145b80 con 0x7f3f501dd1c0 2024-04-18T10:06:51.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f3f500909a0 con 0x7f3f3c0051d0 2024-04-18T10:06:51.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f4ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f501e0770 0x7f3f501e0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f4ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f501e0770 0x7f3f501e0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48856/0 (socket says 172.21.15.62:48856) 2024-04-18T10:06:51.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f4ffff640 1 -- 172.21.15.62:0/1888782064 learned_addr learned my addr 172.21.15.62:0/1888782064 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f54e59640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f3c0051d0 0x7f3f50091800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f5565a640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f501dd1c0 0x7f3f501dd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f4ffff640 1 -- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f3c0051d0 msgr2=0x7f3f50091800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f4ffff640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f3c0051d0 0x7f3f50091800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f4ffff640 1 -- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f501dd1c0 msgr2=0x7f3f501dd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f4ffff640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f501dd1c0 0x7f3f501dd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f4ffff640 1 -- 172.21.15.62:0/1888782064 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f50090c40 con 0x7f3f501e0770 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f54e59640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f3c0051d0 0x7f3f50091800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f4ffff640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f501e0770 0x7f3f501e0bf0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3f38010be0 tx=0x7f3f3800e4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f4d7fa640 1 -- 172.21.15.62:0/1888782064 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f3801b070 con 0x7f3f501e0770 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f4d7fa640 1 -- 172.21.15.62:0/1888782064 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f38011040 con 0x7f3f501e0770 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 -- 172.21.15.62:0/1888782064 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f500931d0 con 0x7f3f501e0770 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f4d7fa640 1 -- 172.21.15.62:0/1888782064 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f380143a0 con 0x7f3f501e0770 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.941+0000 7f3f570bd640 1 -- 172.21.15.62:0/1888782064 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f500726b0 con 0x7f3f501e0770 2024-04-18T10:06:51.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.945+0000 7f3f570bd640 1 -- 172.21.15.62:0/1888782064 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f50068b90 con 0x7f3f501e0770 2024-04-18T10:06:51.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.945+0000 7f3f4d7fa640 1 -- 172.21.15.62:0/1888782064 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3f3801e070 con 0x7f3f501e0770 2024-04-18T10:06:51.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.945+0000 7f3f4d7fa640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3f30068380 0x7f3f3006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:51.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.945+0000 7f3f4d7fa640 1 -- 172.21.15.62:0/1888782064 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f3f38089590 con 0x7f3f501e0770 2024-04-18T10:06:51.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.945+0000 7f3f54e59640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3f30068380 0x7f3f3006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:51.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.945+0000 7f3f54e59640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3f30068380 0x7f3f3006a840 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3f40001600 tx=0x7f3f40009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:51.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:51.949+0000 7f3f4d7fa640 1 -- 172.21.15.62:0/1888782064 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3f38059be0 con 0x7f3f501e0770 2024-04-18T10:06:52.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.149+0000 7f3f570bd640 1 -- 172.21.15.62:0/1888782064 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f500b38a0 con 0x7f3f501e0770 2024-04-18T10:06:52.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.149+0000 7f3f4d7fa640 1 -- 172.21.15.62:0/1888782064 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+3534 (secure 0 0 0) 0x7f3f380594e0 con 0x7f3f501e0770 2024-04-18T10:06:52.150 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:52.150 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4705,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1649467687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1649467687},{"type":"v1","addr":"172.21.15.145:6835","nonce":1649467687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:06:38.304481+0000","modified":"2024-04-18T10:06:41.339980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4932":{"gid":4932,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2971073685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2971073685},{"type":"v1","addr":"172.21.15.62:6833","nonce":2971073685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4932,"qdb_cluster":[4932]},"id":2}]} 2024-04-18T10:06:52.150 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-04-18T10:06:52.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.153+0000 7f3f570bd640 1 -- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3f30068380 msgr2=0x7f3f3006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.153+0000 7f3f570bd640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3f30068380 0x7f3f3006a840 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3f40001600 tx=0x7f3f40009040 comp rx=0 tx=0).stop 2024-04-18T10:06:52.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.153+0000 7f3f570bd640 1 -- 172.21.15.62:0/1888782064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f501e0770 msgr2=0x7f3f501e0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.153+0000 7f3f570bd640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f501e0770 0x7f3f501e0bf0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3f38010be0 tx=0x7f3f3800e4d0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.153+0000 7f3f570bd640 1 -- 172.21.15.62:0/1888782064 shutdown_connections 2024-04-18T10:06:52.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.153+0000 7f3f570bd640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3f30068380 0x7f3f3006a840 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.153+0000 7f3f570bd640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f501dd1c0 0x7f3f501dd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.153+0000 7f3f570bd640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3f501e0770 0x7f3f501e0bf0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.152 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.153+0000 7f3f570bd640 1 --2- 172.21.15.62:0/1888782064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f3c0051d0 0x7f3f50091800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.153+0000 7f3f570bd640 1 -- 172.21.15.62:0/1888782064 >> 172.21.15.62:0/1888782064 conn(0x7f3f5006c0d0 msgr2=0x7f3f500738b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:52.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.153+0000 7f3f570bd640 1 -- 172.21.15.62:0/1888782064 shutdown_connections 2024-04-18T10:06:52.153 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.153+0000 7f3f570bd640 1 -- 172.21.15.62:0/1888782064 wait complete. 2024-04-18T10:06:52.168 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:06:52.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f31274f6640 1 Processor -- start 2024-04-18T10:06:52.292 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f31274f6640 1 -- start start 2024-04-18T10:06:52.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f31274f6640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3120142010 0x7f31201535c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f31274f6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31200b6740 con 0x7f31200b7f70 2024-04-18T10:06:52.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f31274f6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31200b7720 con 0x7f31200b6360 2024-04-18T10:06:52.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f31274f6640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f3120147e40 con 0x7f3120142010 2024-04-18T10:06:52.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3124a91640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f31200b7f70 0x7f3120141900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48708/0 (socket says 172.21.15.62:48708) 2024-04-18T10:06:52.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3124a91640 1 -- 172.21.15.62:0/3234407391 learned_addr learned my addr 172.21.15.62:0/3234407391 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:52.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 954691982 0 0) 0x7f31200b6740 con 0x7f31200b7f70 2024-04-18T10:06:52.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3125a93640 1 --2- 172.21.15.62:0/3234407391 >> v2:172.21.15.145:3300/0 conn(0x7f3120142010 0x7f31201535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f310c003530 con 0x7f31200b7f70 2024-04-18T10:06:52.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1546373045 0 0) 0x7f310c003530 con 0x7f31200b7f70 2024-04-18T10:06:52.295 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 >> v2:172.21.15.145:3300/0 conn(0x7f3120142010 msgr2=0x7f31201535c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 --2- 172.21.15.62:0/3234407391 >> v2:172.21.15.145:3300/0 conn(0x7f3120142010 0x7f31201535c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 >> v1:172.21.15.145:6789/0 conn(0x7f31200b6360 legacy=0x7f3120148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3120148120 con 0x7f31200b7f70 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f3110001680 con 0x7f31200b7f70 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 >> v1:172.21.15.62:6789/0 conn(0x7f31200b7f70 legacy=0x7f3120141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f310c0039c0 0x7f310c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 0x7f310c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f310c0052f0 0x7f310c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f3120147e40 con 0x7f310c004810 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f310c00c1c0 con 0x7f310c0052f0 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f310c00c3c0 con 0x7f310c0039c0 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3124a91640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 0x7f310c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3125a93640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f310c0052f0 0x7f310c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.296 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3125292640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f310c0039c0 0x7f310c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3124a91640 1 -- 172.21.15.62:0/3234407391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f310c0039c0 msgr2=0x7f310c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:52.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3124a91640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f310c0039c0 0x7f310c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3124a91640 1 -- 172.21.15.62:0/3234407391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f310c0052f0 msgr2=0x7f310c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:52.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3124a91640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f310c0052f0 0x7f310c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3124a91640 1 -- 172.21.15.62:0/3234407391 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f310c003530 con 0x7f310c004810 2024-04-18T10:06:52.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3125a93640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f310c0052f0 0x7f310c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:52.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3125a93640 1 -- 172.21.15.62:0/3234407391 reap_dead start 2024-04-18T10:06:52.297 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.293+0000 7f3125292640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f310c0039c0 0x7f310c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:52.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3124a91640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 0x7f310c004c90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f311000c190 tx=0x7f311001cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9502ce9982c8b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110007d00 con 0x7f310c004810 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31100a9020 con 0x7f310c004810 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3117fff640 1 -- 172.21.15.62:0/3234407391 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3110005ae0 con 0x7f310c004810 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 -- 172.21.15.62:0/3234407391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 msgr2=0x7f310c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 0x7f310c004c90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f311000c190 tx=0x7f311001cc40 comp rx=0 tx=0).stop 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 -- 172.21.15.62:0/3234407391 shutdown_connections 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 --2- 172.21.15.62:0/3234407391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 0x7f310c004c90 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 -- 172.21.15.62:0/3234407391 >> 172.21.15.62:0/3234407391 conn(0x7f31200ba070 msgr2=0x7f31200ba480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 -- 172.21.15.62:0/3234407391 shutdown_connections 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 -- 172.21.15.62:0/3234407391 wait complete. 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 Processor -- start 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 -- start start 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 0x7f31201473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f312013c2f0 0x7f3120139bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31201e4420 0x7f31201e48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f31200b50c0 con 0x7f310c004810 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f31200b4f40 con 0x7f312013c2f0 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f31274f6640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f31200b7240 con 0x7f31201e4420 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3125292640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 0x7f31201473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3125292640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 0x7f31201473a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48876/0 (socket says 172.21.15.62:48876) 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3125292640 1 -- 172.21.15.62:0/494991965 learned_addr learned my addr 172.21.15.62:0/494991965 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3124a91640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f312013c2f0 0x7f3120139bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3125a93640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31201e4420 0x7f31201e48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3125292640 1 -- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31201e4420 msgr2=0x7f31201e48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3125292640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31201e4420 0x7f31201e48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3125292640 1 -- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f312013c2f0 msgr2=0x7f3120139bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3125292640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f312013c2f0 0x7f3120139bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3125292640 1 -- 172.21.15.62:0/494991965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312013bbf0 con 0x7f310c004810 2024-04-18T10:06:52.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3125a93640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31201e4420 0x7f31201e48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:52.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.297+0000 7f3125292640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 0x7f31201473a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f311c00d4a0 tx=0x7f311c00d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:52.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.301+0000 7f3115ffb640 1 -- 172.21.15.62:0/494991965 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c005280 con 0x7f310c004810 2024-04-18T10:06:52.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.301+0000 7f3115ffb640 1 -- 172.21.15.62:0/494991965 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f311c005440 con 0x7f310c004810 2024-04-18T10:06:52.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.301+0000 7f31274f6640 1 -- 172.21.15.62:0/494991965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f312013aac0 con 0x7f310c004810 2024-04-18T10:06:52.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.301+0000 7f3115ffb640 1 -- 172.21.15.62:0/494991965 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c007260 con 0x7f310c004810 2024-04-18T10:06:52.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.301+0000 7f31274f6640 1 -- 172.21.15.62:0/494991965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f312013a5e0 con 0x7f310c004810 2024-04-18T10:06:52.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.301+0000 7f31274f6640 1 -- 172.21.15.62:0/494991965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3120145650 con 0x7f310c004810 2024-04-18T10:06:52.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.301+0000 7f3115ffb640 1 -- 172.21.15.62:0/494991965 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f311c019020 con 0x7f310c004810 2024-04-18T10:06:52.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.301+0000 7f3115ffb640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3104068380 0x7f310406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.301+0000 7f3115ffb640 1 -- 172.21.15.62:0/494991965 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f311c086060 con 0x7f310c004810 2024-04-18T10:06:52.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.301+0000 7f3124a91640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3104068380 0x7f310406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.301+0000 7f3124a91640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3104068380 0x7f310406a840 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f31100aa000 tx=0x7f31200b78f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:52.305 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.305+0000 7f3115ffb640 1 -- 172.21.15.62:0/494991965 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f311c0566b0 con 0x7f310c004810 2024-04-18T10:06:52.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.493+0000 7f31274f6640 1 -- 172.21.15.62:0/494991965 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31200b6740 con 0x7f310c004810 2024-04-18T10:06:52.496 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f3115ffb640 1 -- 172.21.15.62:0/494991965 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+3534 (secure 0 0 0) 0x7f311c055fb0 con 0x7f310c004810 2024-04-18T10:06:52.496 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:52.497 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4705,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1649467687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1649467687},{"type":"v1","addr":"172.21.15.145:6835","nonce":1649467687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-18T10:06:38.304481+0000","modified":"2024-04-18T10:06:41.339980+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4932":{"gid":4932,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6833/2971073685","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2971073685},{"type":"v1","addr":"172.21.15.62:6833","nonce":2971073685}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4932,"qdb_cluster":[4932]},"id":2}]} 2024-04-18T10:06:52.497 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 16 2024-04-18T10:06:52.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f31274f6640 1 -- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3104068380 msgr2=0x7f310406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f31274f6640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3104068380 0x7f310406a840 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f31100aa000 tx=0x7f31200b78f0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f31274f6640 1 -- 172.21.15.62:0/494991965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 msgr2=0x7f31201473a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f31274f6640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 0x7f31201473a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f311c00d4a0 tx=0x7f311c00d960 comp rx=0 tx=0).stop 2024-04-18T10:06:52.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f31274f6640 1 -- 172.21.15.62:0/494991965 shutdown_connections 2024-04-18T10:06:52.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f31274f6640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f3104068380 0x7f310406a840 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f31274f6640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31201e4420 0x7f31201e48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f31274f6640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f312013c2f0 0x7f3120139bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f31274f6640 1 --2- 172.21.15.62:0/494991965 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f310c004810 0x7f31201473a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f31274f6640 1 -- 172.21.15.62:0/494991965 >> 172.21.15.62:0/494991965 conn(0x7f31200ba070 msgr2=0x7f3120138b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:52.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f31274f6640 1 -- 172.21.15.62:0/494991965 shutdown_connections 2024-04-18T10:06:52.500 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.497+0000 7f31274f6640 1 -- 172.21.15.62:0/494991965 wait complete. 2024-04-18T10:06:52.514 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:52.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.633+0000 7fe204c79640 1 Processor -- start 2024-04-18T10:06:52.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.633+0000 7fe204c79640 1 -- start start 2024-04-18T10:06:52.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe204c79640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2000b9d90 0x7fe20006d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe204c79640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe20009be30 0x7fe20006d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe204c79640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2000ba170 con 0x7fe20006dc90 2024-04-18T10:06:52.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe204c79640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fe20009cd30 con 0x7fe20009be30 2024-04-18T10:06:52.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe204c79640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fe20014a060 con 0x7fe2000b9d90 2024-04-18T10:06:52.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1f7fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe20009be30 0x7fe20006d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1f7fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe20009be30 0x7fe20006d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48892/0 (socket says 172.21.15.62:48892) 2024-04-18T10:06:52.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1f7fff640 1 -- 172.21.15.62:0/2375100745 learned_addr learned my addr 172.21.15.62:0/2375100745 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:52.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1fe59c640 1 --2- 172.21.15.62:0/2375100745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2000b9d90 0x7fe20006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1f7fff640 1 -- 172.21.15.62:0/2375100745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2000b9d90 msgr2=0x7fe20006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:52.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1f7fff640 1 --2- 172.21.15.62:0/2375100745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2000b9d90 0x7fe20006d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1f7fff640 1 -- 172.21.15.62:0/2375100745 >> v1:172.21.15.145:6789/0 conn(0x7fe20006dc90 legacy=0x7fe200144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1f7fff640 1 -- 172.21.15.62:0/2375100745 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe200149860 con 0x7fe20009be30 2024-04-18T10:06:52.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1fe59c640 1 --2- 172.21.15.62:0/2375100745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2000b9d90 0x7fe20006d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:52.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1f7fff640 1 --2- 172.21.15.62:0/2375100745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe20009be30 0x7fe20006d750 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009870 tx=0x7fe1f001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d590cb76d41f013 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:52.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1fdd9b640 1 -- 172.21.15.62:0/2375100745 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f00b0070 con 0x7fe20009be30 2024-04-18T10:06:52.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1fdd9b640 1 -- 172.21.15.62:0/2375100745 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1f00ab600 con 0x7fe20009be30 2024-04-18T10:06:52.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.637+0000 7fe1fdd9b640 1 -- 172.21.15.62:0/2375100745 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f00ab9d0 con 0x7fe20009be30 2024-04-18T10:06:52.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 -- 172.21.15.62:0/2375100745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe20009be30 msgr2=0x7fe20006d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 --2- 172.21.15.62:0/2375100745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe20009be30 0x7fe20006d750 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009870 tx=0x7fe1f001c440 comp rx=0 tx=0).stop 2024-04-18T10:06:52.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 -- 172.21.15.62:0/2375100745 shutdown_connections 2024-04-18T10:06:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 --2- 172.21.15.62:0/2375100745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe20009be30 0x7fe20006d750 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 --2- 172.21.15.62:0/2375100745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2000b9d90 0x7fe20006d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 -- 172.21.15.62:0/2375100745 >> 172.21.15.62:0/2375100745 conn(0x7fe20006c0d0 msgr2=0x7fe2000b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 -- 172.21.15.62:0/2375100745 shutdown_connections 2024-04-18T10:06:52.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 -- 172.21.15.62:0/2375100745 wait complete. 2024-04-18T10:06:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 Processor -- start 2024-04-18T10:06:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 -- start start 2024-04-18T10:06:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe20006dc90 0x7fe200071b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe20009be30 0x7fe200070a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2000b9d90 0x7fe2000b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe200145fb0 con 0x7fe20009be30 2024-04-18T10:06:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fe200145e30 con 0x7fe20006dc90 2024-04-18T10:06:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fe200146130 con 0x7fe2000b9d90 2024-04-18T10:06:52.643 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1f7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe20009be30 0x7fe200070a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1f7fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe20009be30 0x7fe200070a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48906/0 (socket says 172.21.15.62:48906) 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1f7fff640 1 -- 172.21.15.62:0/3779279868 learned_addr learned my addr 172.21.15.62:0/3779279868 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1fed9d640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2000b9d90 0x7fe2000b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1fe59c640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe20006dc90 0x7fe200071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1f7fff640 1 -- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2000b9d90 msgr2=0x7fe2000b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1f7fff640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2000b9d90 0x7fe2000b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1f7fff640 1 -- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe20006dc90 msgr2=0x7fe200071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1f7fff640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe20006dc90 0x7fe200071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1f7fff640 1 -- 172.21.15.62:0/3779279868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe200072b40 con 0x7fe20009be30 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1f7fff640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe20009be30 0x7fe200070a10 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00ae040 tx=0x7fe1f00191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1f6ffd640 1 -- 172.21.15.62:0/3779279868 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f00b0040 con 0x7fe20009be30 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1f6ffd640 1 -- 172.21.15.62:0/3779279868 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1f00b4440 con 0x7fe20009be30 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe1f6ffd640 1 -- 172.21.15.62:0/3779279868 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f00b4830 con 0x7fe20009be30 2024-04-18T10:06:52.644 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 -- 172.21.15.62:0/3779279868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe200072360 con 0x7fe20009be30 2024-04-18T10:06:52.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 -- 172.21.15.62:0/3779279868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2000703f0 con 0x7fe20009be30 2024-04-18T10:06:52.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.641+0000 7fe204c79640 1 -- 172.21.15.62:0/3779279868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe200068b90 con 0x7fe20009be30 2024-04-18T10:06:52.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.645+0000 7fe1f6ffd640 1 -- 172.21.15.62:0/3779279868 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe1f00bb020 con 0x7fe20009be30 2024-04-18T10:06:52.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.645+0000 7fe1f6ffd640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe1e0068380 0x7fe1e006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.645+0000 7fe1f6ffd640 1 -- 172.21.15.62:0/3779279868 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7fe1f011fe10 con 0x7fe20009be30 2024-04-18T10:06:52.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.645+0000 7fe1fe59c640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe1e0068380 0x7fe1e006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.645+0000 7fe1fe59c640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe1e0068380 0x7fe1e006a840 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8006fd0 tx=0x7fe1e8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:52.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.649+0000 7fe1f6ffd640 1 -- 172.21.15.62:0/3779279868 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe1f00c2020 con 0x7fe20009be30 2024-04-18T10:06:52.811 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.809+0000 7fe204c79640 1 -- 172.21.15.62:0/3779279868 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe2000b38f0 con 0x7fe20009be30 2024-04-18T10:06:52.814 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.813+0000 7fe1f6ffd640 1 -- 172.21.15.62:0/3779279868 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) ==== 74+0+16657 (secure 0 0 0) 0x7fe1f0016150 con 0x7fe20009be30 2024-04-18T10:06:52.814 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:52.814 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":30,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:39.937405+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-04-18T10:06:36.802689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"29","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-04-18T10:06:37.269237+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.813+0000 7fe204c79640 1 -- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe1e0068380 msgr2=0x7fe1e006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.813+0000 7fe204c79640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe1e0068380 0x7fe1e006a840 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8006fd0 tx=0x7fe1e8008040 comp rx=0 tx=0).stop 2024-04-18T10:06:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.813+0000 7fe204c79640 1 -- 172.21.15.62:0/3779279868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe20009be30 msgr2=0x7fe200070a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.813+0000 7fe204c79640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe20009be30 0x7fe200070a10 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00ae040 tx=0x7fe1f00191e0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.817+0000 7fe204c79640 1 -- 172.21.15.62:0/3779279868 shutdown_connections 2024-04-18T10:06:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.817+0000 7fe204c79640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe1e0068380 0x7fe1e006a840 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.817+0000 7fe204c79640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2000b9d90 0x7fe2000b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.817+0000 7fe204c79640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe20009be30 0x7fe200070a10 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.817+0000 7fe204c79640 1 --2- 172.21.15.62:0/3779279868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe20006dc90 0x7fe200071b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.817+0000 7fe204c79640 1 -- 172.21.15.62:0/3779279868 >> 172.21.15.62:0/3779279868 conn(0x7fe20006c0d0 msgr2=0x7fe2000afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:52.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.817+0000 7fe204c79640 1 -- 172.21.15.62:0/3779279868 shutdown_connections 2024-04-18T10:06:52.817 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.817+0000 7fe204c79640 1 -- 172.21.15.62:0/3779279868 wait complete. 2024-04-18T10:06:52.833 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-18T10:06:52.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84e0a45640 1 Processor -- start 2024-04-18T10:06:52.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84e0a45640 1 -- start start 2024-04-18T10:06:52.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84e0a45640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f84dc0b6300 0x7f84dc14b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84e0a45640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f84dc1486d0 0x7f84dc141e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84e0a45640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84dc0b74e0 con 0x7f84dc148240 2024-04-18T10:06:52.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84e0a45640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f84dc1423c0 con 0x7f84dc0b6300 2024-04-18T10:06:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84e0a45640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f84dc0b7750 con 0x7f84dc1486d0 2024-04-18T10:06:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84d9d9b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f84dc148240 0x7f84dc141770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48720/0 (socket says 172.21.15.62:48720) 2024-04-18T10:06:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84da59c640 1 --2- 172.21.15.62:0/1018779760 >> v2:172.21.15.62:3300/0 conn(0x7f84dc0b6300 0x7f84dc14b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84d9d9b640 1 -- 172.21.15.62:0/1018779760 learned_addr learned my addr 172.21.15.62:0/1018779760 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84dad9d640 1 --2- 172.21.15.62:0/1018779760 >> v2:172.21.15.145:3300/0 conn(0x7f84dc1486d0 0x7f84dc141e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84da59c640 1 -- 172.21.15.62:0/1018779760 >> v2:172.21.15.145:3300/0 conn(0x7f84dc1486d0 msgr2=0x7f84dc141e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84da59c640 1 --2- 172.21.15.62:0/1018779760 >> v2:172.21.15.145:3300/0 conn(0x7f84dc1486d0 0x7f84dc141e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84da59c640 1 -- 172.21.15.62:0/1018779760 >> v1:172.21.15.62:6789/0 conn(0x7f84dc148240 legacy=0x7f84dc141770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84da59c640 1 -- 172.21.15.62:0/1018779760 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84dc147d90 con 0x7f84dc0b6300 2024-04-18T10:06:52.963 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.961+0000 7f84dad9d640 1 --2- 172.21.15.62:0/1018779760 >> v2:172.21.15.145:3300/0 conn(0x7f84dc1486d0 0x7f84dc141e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:52.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84da59c640 1 --2- 172.21.15.62:0/1018779760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84dc0b6300 0x7f84dc14b750 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f84dc14bc90 tx=0x7f84cc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6feaa547ec6959db server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:52.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84d959a640 1 -- 172.21.15.62:0/1018779760 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc0ab680 con 0x7f84dc0b6300 2024-04-18T10:06:52.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84d959a640 1 -- 172.21.15.62:0/1018779760 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84cc0b0070 con 0x7f84dc0b6300 2024-04-18T10:06:52.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84d959a640 1 -- 172.21.15.62:0/1018779760 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc0aba40 con 0x7f84dc0b6300 2024-04-18T10:06:52.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84e0a45640 1 -- 172.21.15.62:0/1018779760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84dc0b6300 msgr2=0x7f84dc14b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84e0a45640 1 --2- 172.21.15.62:0/1018779760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84dc0b6300 0x7f84dc14b750 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f84dc14bc90 tx=0x7f84cc01c480 comp rx=0 tx=0).stop 2024-04-18T10:06:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84e0a45640 1 -- 172.21.15.62:0/1018779760 shutdown_connections 2024-04-18T10:06:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84e0a45640 1 --2- 172.21.15.62:0/1018779760 >> v2:172.21.15.145:3300/0 conn(0x7f84dc1486d0 0x7f84dc141e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84e0a45640 1 --2- 172.21.15.62:0/1018779760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84dc0b6300 0x7f84dc14b750 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84e0a45640 1 -- 172.21.15.62:0/1018779760 >> 172.21.15.62:0/1018779760 conn(0x7f84dc068090 msgr2=0x7f84dc0bd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84e0a45640 1 -- 172.21.15.62:0/1018779760 shutdown_connections 2024-04-18T10:06:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84e0a45640 1 -- 172.21.15.62:0/1018779760 wait complete. 2024-04-18T10:06:52.966 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84e0a45640 1 Processor -- start 2024-04-18T10:06:52.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.965+0000 7f84e0a45640 1 -- start start 2024-04-18T10:06:52.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84e0a45640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84dc0b6300 0x7f84dc1e0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84e0a45640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84dc148240 0x7f84dc140ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84e0a45640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84dc1486d0 0x7f84dc14d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84e0a45640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc154e20 con 0x7f84dc148240 2024-04-18T10:06:52.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84e0a45640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc14c060 con 0x7f84dc0b6300 2024-04-18T10:06:52.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84e0a45640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f84dc154fa0 con 0x7f84dc1486d0 2024-04-18T10:06:52.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84d9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84dc148240 0x7f84dc140ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84d9d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84dc148240 0x7f84dc140ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48926/0 (socket says 172.21.15.62:48926) 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84d9d9b640 1 -- 172.21.15.62:0/280739 learned_addr learned my addr 172.21.15.62:0/280739 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84da59c640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84dc0b6300 0x7f84dc1e0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84dad9d640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84dc1486d0 0x7f84dc14d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84d9d9b640 1 -- 172.21.15.62:0/280739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84dc1486d0 msgr2=0x7f84dc14d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84d9d9b640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84dc1486d0 0x7f84dc14d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84d9d9b640 1 -- 172.21.15.62:0/280739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84dc0b6300 msgr2=0x7f84dc1e0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84d9d9b640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84dc0b6300 0x7f84dc1e0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84d9d9b640 1 -- 172.21.15.62:0/280739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84dc14cc30 con 0x7f84dc148240 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84da59c640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84dc0b6300 0x7f84dc1e0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84d9d9b640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84dc148240 0x7f84dc140ba0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f84d000c420 tx=0x7f84d000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84caffd640 1 -- 172.21.15.62:0/280739 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d0017070 con 0x7f84dc148240 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84caffd640 1 -- 172.21.15.62:0/280739 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84d0005cf0 con 0x7f84dc148240 2024-04-18T10:06:52.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84e0a45640 1 -- 172.21.15.62:0/280739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84dc14e960 con 0x7f84dc148240 2024-04-18T10:06:52.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84caffd640 1 -- 172.21.15.62:0/280739 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d00072f0 con 0x7f84dc148240 2024-04-18T10:06:52.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84e0a45640 1 -- 172.21.15.62:0/280739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84dc145650 con 0x7f84dc148240 2024-04-18T10:06:52.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84e0a45640 1 -- 172.21.15.62:0/280739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84dc073b40 con 0x7f84dc148240 2024-04-18T10:06:52.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84caffd640 1 -- 172.21.15.62:0/280739 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f84d001a070 con 0x7f84dc148240 2024-04-18T10:06:52.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84caffd640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f84bc068380 0x7f84bc06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:52.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84caffd640 1 -- 172.21.15.62:0/280739 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5709+0+0 (secure 0 0 0) 0x7f84d0085ce0 con 0x7f84dc148240 2024-04-18T10:06:52.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.969+0000 7f84da59c640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f84bc068380 0x7f84bc06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:52.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.973+0000 7f84da59c640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f84bc068380 0x7f84bc06a840 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f84cc0af000 tx=0x7f84cc0a7000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:52.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:52.977+0000 7f84caffd640 1 -- 172.21.15.62:0/280739 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f84d0056330 con 0x7f84dc148240 2024-04-18T10:06:53.175 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.173+0000 7f84e0a45640 1 -- 172.21.15.62:0/280739 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f84dc068b20 con 0x7f84dc148240 2024-04-18T10:06:53.176 INFO:tasks.ceph.mon.a.smithi062.stderr:2024-04-18T10:06:53.177+0000 7f0d8301a640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-18T10:06:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.177+0000 7f84caffd640 1 -- 172.21.15.62:0/280739 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7f84d0055c30 con 0x7f84dc148240 2024-04-18T10:06:53.178 INFO:teuthology.orchestra.run.smithi062.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-18T10:06:53.180 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.181+0000 7f84e0a45640 1 -- 172.21.15.62:0/280739 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f84bc068380 msgr2=0x7f84bc06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.181+0000 7f84e0a45640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f84bc068380 0x7f84bc06a840 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f84cc0af000 tx=0x7f84cc0a7000 comp rx=0 tx=0).stop 2024-04-18T10:06:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.181+0000 7f84e0a45640 1 -- 172.21.15.62:0/280739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84dc148240 msgr2=0x7f84dc140ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:53.181 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.181+0000 7f84e0a45640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84dc148240 0x7f84dc140ba0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f84d000c420 tx=0x7f84d000c8e0 comp rx=0 tx=0).stop 2024-04-18T10:06:53.184 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.185+0000 7f84e0a45640 1 -- 172.21.15.62:0/280739 shutdown_connections 2024-04-18T10:06:53.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.185+0000 7f84e0a45640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f84bc068380 0x7f84bc06a840 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:53.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.185+0000 7f84e0a45640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84dc1486d0 0x7f84dc14d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:53.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.185+0000 7f84e0a45640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84dc148240 0x7f84dc140ba0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:53.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.185+0000 7f84e0a45640 1 --2- 172.21.15.62:0/280739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f84dc0b6300 0x7f84dc1e0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:53.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.185+0000 7f84e0a45640 1 -- 172.21.15.62:0/280739 >> 172.21.15.62:0/280739 conn(0x7f84dc068090 msgr2=0x7f84dc147230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:53.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.185+0000 7f84e0a45640 1 -- 172.21.15.62:0/280739 shutdown_connections 2024-04-18T10:06:53.185 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.185+0000 7f84e0a45640 1 -- 172.21.15.62:0/280739 wait complete. 2024-04-18T10:06:53.186 INFO:tasks.ceph.mds.a.smithi062.stderr: -2436> 2024-04-18T10:06:42.897+0000 7f274e1b5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-18T10:06:53.202 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-18T10:06:53.220 INFO:tasks.ceph.mds.a.smithi062.stdout:starting mds.a at 2024-04-18T10:06:53.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb9b3f9640 1 Processor -- start 2024-04-18T10:06:53.321 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb9b3f9640 1 -- start start 2024-04-18T10:06:53.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb9b3f9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb940b83e0 0x7fbb940b8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb9b3f9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbb941468b0 0x7fbb9413fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb9b3f9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb940b87c0 con 0x7fbb9413fa20 2024-04-18T10:06:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb9b3f9640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fbb940b99b0 con 0x7fbb941468b0 2024-04-18T10:06:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb9b3f9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fbb940ba820 con 0x7fbb940b83e0 2024-04-18T10:06:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb99996640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbb941468b0 0x7fbb9413fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb99996640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fbb941468b0 0x7fbb9413fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48948/0 (socket says 172.21.15.62:48948) 2024-04-18T10:06:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb99996640 1 -- 172.21.15.62:0/2410320511 learned_addr learned my addr 172.21.15.62:0/2410320511 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb99195640 1 --2- 172.21.15.62:0/2410320511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb940b83e0 0x7fbb940b8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb99996640 1 -- 172.21.15.62:0/2410320511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb940b83e0 msgr2=0x7fbb940b8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb99996640 1 --2- 172.21.15.62:0/2410320511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb940b83e0 0x7fbb940b8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb99996640 1 -- 172.21.15.62:0/2410320511 >> v1:172.21.15.145:6789/0 conn(0x7fbb9413fa20 legacy=0x7fbb941461a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:53.323 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.321+0000 7fbb99996640 1 -- 172.21.15.62:0/2410320511 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb940b96d0 con 0x7fbb941468b0 2024-04-18T10:06:53.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb99996640 1 --2- 172.21.15.62:0/2410320511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb941468b0 0x7fbb9413fe40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbb90009870 tx=0x7fbb9001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ec54fa55ccb178e server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:53.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb8bfff640 1 -- 172.21.15.62:0/2410320511 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb900b2070 con 0x7fbb941468b0 2024-04-18T10:06:53.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb8bfff640 1 -- 172.21.15.62:0/2410320511 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb900a9070 con 0x7fbb941468b0 2024-04-18T10:06:53.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb8bfff640 1 -- 172.21.15.62:0/2410320511 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb900ad880 con 0x7fbb941468b0 2024-04-18T10:06:53.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/2410320511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb941468b0 msgr2=0x7fbb9413fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:53.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 --2- 172.21.15.62:0/2410320511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb941468b0 0x7fbb9413fe40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fbb90009870 tx=0x7fbb9001c560 comp rx=0 tx=0).stop 2024-04-18T10:06:53.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/2410320511 shutdown_connections 2024-04-18T10:06:53.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 --2- 172.21.15.62:0/2410320511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb941468b0 0x7fbb9413fe40 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:53.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 --2- 172.21.15.62:0/2410320511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb940b83e0 0x7fbb940b8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:53.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/2410320511 >> 172.21.15.62:0/2410320511 conn(0x7fbb9409d4e0 msgr2=0x7fbb9409d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:53.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/2410320511 shutdown_connections 2024-04-18T10:06:53.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/2410320511 wait complete. 2024-04-18T10:06:53.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 Processor -- start 2024-04-18T10:06:53.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 -- start start 2024-04-18T10:06:53.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb940b83e0 0x7fbb94146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:53.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb9413fa20 0x7fbb941de6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:53.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb941468b0 0x7fbb941dec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:53.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fbb94152fc0 con 0x7fbb940b83e0 2024-04-18T10:06:53.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fbb94152e40 con 0x7fbb941468b0 2024-04-18T10:06:53.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb9b3f9640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fbb94153140 con 0x7fbb9413fa20 2024-04-18T10:06:53.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb99195640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb940b83e0 0x7fbb94146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:53.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb99195640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb940b83e0 0x7fbb94146f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48958/0 (socket says 172.21.15.62:48958) 2024-04-18T10:06:53.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb99195640 1 -- 172.21.15.62:0/997756792 learned_addr learned my addr 172.21.15.62:0/997756792 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:53.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb99996640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb941468b0 0x7fbb941dec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:53.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb98994640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb9413fa20 0x7fbb941de6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:53.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb99195640 1 -- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb9413fa20 msgr2=0x7fbb941de6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:53.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb99195640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb9413fa20 0x7fbb941de6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:53.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb99195640 1 -- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb941468b0 msgr2=0x7fbb941dec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:53.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb99195640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb941468b0 0x7fbb941dec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:53.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb99195640 1 -- 172.21.15.62:0/997756792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9413f360 con 0x7fbb940b83e0 2024-04-18T10:06:53.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.325+0000 7fbb99996640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb941468b0 0x7fbb941dec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:53.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.329+0000 7fbb99195640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb940b83e0 0x7fbb94146f30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009dd0 tx=0x7fbb8c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:53.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.329+0000 7fbb89ffb640 1 -- 172.21.15.62:0/997756792 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c018020 con 0x7fbb940b83e0 2024-04-18T10:06:53.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.329+0000 7fbb89ffb640 1 -- 172.21.15.62:0/997756792 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb8c014480 con 0x7fbb940b83e0 2024-04-18T10:06:53.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.329+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/997756792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb94148850 con 0x7fbb940b83e0 2024-04-18T10:06:53.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.329+0000 7fbb89ffb640 1 -- 172.21.15.62:0/997756792 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c014870 con 0x7fbb940b83e0 2024-04-18T10:06:53.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.329+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/997756792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb941de1b0 con 0x7fbb940b83e0 2024-04-18T10:06:53.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.329+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/997756792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb941de3b0 con 0x7fbb940b83e0 2024-04-18T10:06:53.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.329+0000 7fbb89ffb640 1 -- 172.21.15.62:0/997756792 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbb8c01b020 con 0x7fbb940b83e0 2024-04-18T10:06:53.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.329+0000 7fbb89ffb640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbb78068380 0x7fbb7806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:53.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.329+0000 7fbb89ffb640 1 -- 172.21.15.62:0/997756792 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5795+0+0 (secure 0 0 0) 0x7fbb8c0878a0 con 0x7fbb940b83e0 2024-04-18T10:06:53.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.329+0000 7fbb98994640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbb78068380 0x7fbb7806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:53.332 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.329+0000 7fbb98994640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbb78068380 0x7fbb7806a840 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbb940b7300 tx=0x7fbb84009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:53.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.333+0000 7fbb89ffb640 1 -- 172.21.15.62:0/997756792 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbb941de3b0 con 0x7fbb940b83e0 2024-04-18T10:06:53.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:53.525+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/997756792 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fbb94069230 con 0x7fbb940b83e0 2024-04-18T10:06:54.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.225+0000 7fbb89ffb640 1 -- 172.21.15.62:0/997756792 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7fbb8c022030 con 0x7fbb940b83e0 2024-04-18T10:06:54.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.229+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbb78068380 msgr2=0x7fbb7806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:54.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.229+0000 7fbb9b3f9640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbb78068380 0x7fbb7806a840 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbb940b7300 tx=0x7fbb84009040 comp rx=0 tx=0).stop 2024-04-18T10:06:54.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.229+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/997756792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb940b83e0 msgr2=0x7fbb94146f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:54.229 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.229+0000 7fbb9b3f9640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb940b83e0 0x7fbb94146f30 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009dd0 tx=0x7fbb8c00c490 comp rx=0 tx=0).stop 2024-04-18T10:06:54.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.229+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/997756792 shutdown_connections 2024-04-18T10:06:54.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.229+0000 7fbb9b3f9640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fbb78068380 0x7fbb7806a840 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:54.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.229+0000 7fbb9b3f9640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb941468b0 0x7fbb941dec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:54.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.229+0000 7fbb9b3f9640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb9413fa20 0x7fbb941de6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:54.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.229+0000 7fbb9b3f9640 1 --2- 172.21.15.62:0/997756792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb940b83e0 0x7fbb94146f30 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:54.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.229+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/997756792 >> 172.21.15.62:0/997756792 conn(0x7fbb9409d4e0 msgr2=0x7fbb94074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:54.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.229+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/997756792 shutdown_connections 2024-04-18T10:06:54.230 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.229+0000 7fbb9b3f9640 1 -- 172.21.15.62:0/997756792 wait complete. 2024-04-18T10:06:54.251 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-18T10:06:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64eb14f640 1 Processor -- start 2024-04-18T10:06:54.425 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64eb14f640 1 -- start start 2024-04-18T10:06:54.426 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64eb14f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f64e414c270 0x7f64e41501f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64eb14f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64e40736b0 0x7f64e4073b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64eb14f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64e4074080 con 0x7f64e40b9d90 2024-04-18T10:06:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64eb14f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f64e4150730 con 0x7f64e414c270 2024-04-18T10:06:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64eb14f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f64e414fd60 con 0x7f64e40736b0 2024-04-18T10:06:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64e96ec640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64e40736b0 0x7f64e4073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64e3fff640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f64e414c270 0x7f64e41501f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64e96ec640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64e40736b0 0x7f64e4073b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.62:46904/0 (socket says 172.21.15.62:46904) 2024-04-18T10:06:54.427 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64e96ec640 1 -- 172.21.15.62:0/3476284774 learned_addr learned my addr 172.21.15.62:0/3476284774 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64e96ec640 1 -- 172.21.15.62:0/3476284774 >> v2:172.21.15.145:3300/0 conn(0x7f64e414c270 msgr2=0x7f64e41501f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64e96ec640 1 --2- 172.21.15.62:0/3476284774 >> v2:172.21.15.145:3300/0 conn(0x7f64e414c270 0x7f64e41501f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64e96ec640 1 -- 172.21.15.62:0/3476284774 >> v1:172.21.15.145:6789/0 conn(0x7f64e40b9d90 legacy=0x7f64e409db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64e96ec640 1 -- 172.21.15.62:0/3476284774 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e40ba170 con 0x7f64e40736b0 2024-04-18T10:06:54.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.425+0000 7f64e3fff640 1 --2- 172.21.15.62:0/3476284774 >> v2:172.21.15.145:3300/0 conn(0x7f64e414c270 0x7f64e41501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:06:54.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.429+0000 7f64e96ec640 1 --2- 172.21.15.62:0/3476284774 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64e40736b0 0x7f64e4073b40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f64dc009870 tx=0x7f64dc01c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90f178a9768dfdb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:54.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.429+0000 7f64e37fe640 1 -- 172.21.15.62:0/3476284774 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc0b0070 con 0x7f64e40736b0 2024-04-18T10:06:54.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.429+0000 7f64e37fe640 1 -- 172.21.15.62:0/3476284774 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64dc0ab600 con 0x7f64e40736b0 2024-04-18T10:06:54.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.429+0000 7f64e37fe640 1 -- 172.21.15.62:0/3476284774 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64dc0ab9d0 con 0x7f64e40736b0 2024-04-18T10:06:54.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.429+0000 7f64eb14f640 1 -- 172.21.15.62:0/3476284774 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64e40736b0 msgr2=0x7f64e4073b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.429+0000 7f64eb14f640 1 --2- 172.21.15.62:0/3476284774 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64e40736b0 0x7f64e4073b40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f64dc009870 tx=0x7f64dc01c4d0 comp rx=0 tx=0).stop 2024-04-18T10:06:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.429+0000 7f64eb14f640 1 -- 172.21.15.62:0/3476284774 shutdown_connections 2024-04-18T10:06:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.429+0000 7f64eb14f640 1 --2- 172.21.15.62:0/3476284774 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64e40736b0 0x7f64e4073b40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.429+0000 7f64eb14f640 1 --2- 172.21.15.62:0/3476284774 >> v2:172.21.15.145:3300/0 conn(0x7f64e414c270 0x7f64e41501f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.429+0000 7f64eb14f640 1 -- 172.21.15.62:0/3476284774 >> 172.21.15.62:0/3476284774 conn(0x7f64e406c0d0 msgr2=0x7f64e40b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.429+0000 7f64eb14f640 1 -- 172.21.15.62:0/3476284774 shutdown_connections 2024-04-18T10:06:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.429+0000 7f64eb14f640 1 -- 172.21.15.62:0/3476284774 wait complete. 2024-04-18T10:06:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64eb14f640 1 Processor -- start 2024-04-18T10:06:54.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64eb14f640 1 -- start start 2024-04-18T10:06:54.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64eb14f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64e40736b0 0x7f64e41405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:54.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64eb14f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64e40b9d90 0x7f64e413def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:54.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64eb14f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64e414c270 0x7f64e41e8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:54.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64eb14f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f64e41594b0 con 0x7f64e40b9d90 2024-04-18T10:06:54.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64eb14f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f64e4159330 con 0x7f64e40736b0 2024-04-18T10:06:54.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64eb14f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f64e4159630 con 0x7f64e414c270 2024-04-18T10:06:54.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64e40b9d90 0x7f64e413def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:54.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e3fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64e40b9d90 0x7f64e413def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48960/0 (socket says 172.21.15.62:48960) 2024-04-18T10:06:54.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e3fff640 1 -- 172.21.15.62:0/2933201609 learned_addr learned my addr 172.21.15.62:0/2933201609 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:54.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e8eeb640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64e40736b0 0x7f64e41405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:54.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e96ec640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64e414c270 0x7f64e41e8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:54.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e3fff640 1 -- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64e414c270 msgr2=0x7f64e41e8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:54.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e3fff640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64e414c270 0x7f64e41e8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:54.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e3fff640 1 -- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64e40736b0 msgr2=0x7f64e41405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:54.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e3fff640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64e40736b0 0x7f64e41405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:54.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e3fff640 1 -- 172.21.15.62:0/2933201609 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e40b43b0 con 0x7f64e40b9d90 2024-04-18T10:06:54.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e3fff640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64e40b9d90 0x7f64e413def0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009ec0 tx=0x7f64cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:54.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e17fa640 1 -- 172.21.15.62:0/2933201609 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc018020 con 0x7f64e40b9d90 2024-04-18T10:06:54.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e17fa640 1 -- 172.21.15.62:0/2933201609 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64cc014680 con 0x7f64e40b9d90 2024-04-18T10:06:54.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64eb14f640 1 -- 172.21.15.62:0/2933201609 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64e4140b80 con 0x7f64e40b9d90 2024-04-18T10:06:54.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e17fa640 1 -- 172.21.15.62:0/2933201609 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc014a50 con 0x7f64e40b9d90 2024-04-18T10:06:54.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64eb14f640 1 -- 172.21.15.62:0/2933201609 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64e4141c90 con 0x7f64e40b9d90 2024-04-18T10:06:54.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64eb14f640 1 -- 172.21.15.62:0/2933201609 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64e4068ca0 con 0x7f64e40b9d90 2024-04-18T10:06:54.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.433+0000 7f64e17fa640 1 -- 172.21.15.62:0/2933201609 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f64cc01b020 con 0x7f64e40b9d90 2024-04-18T10:06:54.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.437+0000 7f64e17fa640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f64c40683b0 0x7f64c406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:54.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.437+0000 7f64e17fa640 1 -- 172.21.15.62:0/2933201609 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5795+0+0 (secure 0 0 0) 0x7f64cc0875d0 con 0x7f64e40b9d90 2024-04-18T10:06:54.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.437+0000 7f64e8eeb640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f64c40683b0 0x7f64c406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:54.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.437+0000 7f64e8eeb640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f64c40683b0 0x7f64c406a870 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f64d4001810 tx=0x7f64d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:54.440 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.441+0000 7f64e17fa640 1 -- 172.21.15.62:0/2933201609 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f64cc022030 con 0x7f64e40b9d90 2024-04-18T10:06:54.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:54.605+0000 7f64eb14f640 1 -- 172.21.15.62:0/2933201609 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f64e40b4980 con 0x7f64e40b9d90 2024-04-18T10:06:55.235 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.233+0000 7f64e17fa640 1 -- 172.21.15.62:0/2933201609 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v32) ==== 181+0+0 (secure 0 0 0) 0x7f64cc010040 con 0x7f64e40b9d90 2024-04-18T10:06:55.236 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_metadata' removed 2024-04-18T10:06:55.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.241+0000 7f64eb14f640 1 -- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f64c40683b0 msgr2=0x7f64c406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:55.241 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.241+0000 7f64eb14f640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f64c40683b0 0x7f64c406a870 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f64d4001810 tx=0x7f64d4009040 comp rx=0 tx=0).stop 2024-04-18T10:06:55.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.241+0000 7f64eb14f640 1 -- 172.21.15.62:0/2933201609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64e40b9d90 msgr2=0x7f64e413def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:55.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.241+0000 7f64eb14f640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64e40b9d90 0x7f64e413def0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009ec0 tx=0x7f64cc00c490 comp rx=0 tx=0).stop 2024-04-18T10:06:55.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.241+0000 7f64eb14f640 1 -- 172.21.15.62:0/2933201609 shutdown_connections 2024-04-18T10:06:55.242 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.241+0000 7f64eb14f640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f64c40683b0 0x7f64c406a870 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:55.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.241+0000 7f64eb14f640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64e414c270 0x7f64e41e8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:55.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.241+0000 7f64eb14f640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64e40b9d90 0x7f64e413def0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:55.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.241+0000 7f64eb14f640 1 --2- 172.21.15.62:0/2933201609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64e40736b0 0x7f64e41405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:55.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.241+0000 7f64eb14f640 1 -- 172.21.15.62:0/2933201609 >> 172.21.15.62:0/2933201609 conn(0x7f64e406c0d0 msgr2=0x7f64e4074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:55.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.241+0000 7f64eb14f640 1 -- 172.21.15.62:0/2933201609 shutdown_connections 2024-04-18T10:06:55.243 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.241+0000 7f64eb14f640 1 -- 172.21.15.62:0/2933201609 wait complete. 2024-04-18T10:06:55.267 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-18T10:06:55.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cf2bd8640 1 Processor -- start 2024-04-18T10:06:55.397 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cf2bd8640 1 -- start start 2024-04-18T10:06:55.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cf2bd8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cec067a70 0x7f9cec073370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:55.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cf2bd8640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9cec072ec0 0x7f9cec070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:55.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cf2bd8640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9cec0711d0 0x7f9cec148040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:55.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cf2bd8640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f9cec067e50 con 0x7f9cec072ec0 2024-04-18T10:06:55.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cf2bd8640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f9cec0723a0 con 0x7f9cec0711d0 2024-04-18T10:06:55.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cf2bd8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f9cec0b0ee0 con 0x7f9cec067a70 2024-04-18T10:06:55.398 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cebfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9cec072ec0 0x7f9cec070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cebfff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9cec072ec0 0x7f9cec070c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48974/0 (socket says 172.21.15.62:48974) 2024-04-18T10:06:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cebfff640 1 -- 172.21.15.62:0/4130882867 learned_addr learned my addr 172.21.15.62:0/4130882867 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cf1175640 1 --2- 172.21.15.62:0/4130882867 >> v2:172.21.15.145:3300/0 conn(0x7f9cec0711d0 0x7f9cec148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cf0974640 1 --2- 172.21.15.62:0/4130882867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cec067a70 0x7f9cec073370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cebfff640 1 -- 172.21.15.62:0/4130882867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cec067a70 msgr2=0x7f9cec073370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cebfff640 1 --2- 172.21.15.62:0/4130882867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cec067a70 0x7f9cec073370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cebfff640 1 -- 172.21.15.62:0/4130882867 >> v2:172.21.15.145:3300/0 conn(0x7f9cec0711d0 msgr2=0x7f9cec148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cebfff640 1 --2- 172.21.15.62:0/4130882867 >> v2:172.21.15.145:3300/0 conn(0x7f9cec0711d0 0x7f9cec148040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:55.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.397+0000 7f9cebfff640 1 -- 172.21.15.62:0/4130882867 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec072ba0 con 0x7f9cec072ec0 2024-04-18T10:06:55.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cebfff640 1 --2- 172.21.15.62:0/4130882867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec072ec0 0x7f9cec070c90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009870 tx=0x7f9cd801c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a772ab8014fd42b server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:55.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9ceb7fe640 1 -- 172.21.15.62:0/4130882867 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cd80b0070 con 0x7f9cec072ec0 2024-04-18T10:06:55.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9ceb7fe640 1 -- 172.21.15.62:0/4130882867 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cd80ab600 con 0x7f9cec072ec0 2024-04-18T10:06:55.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9ceb7fe640 1 -- 172.21.15.62:0/4130882867 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cec0b0ca0 con 0x7f9cec072ec0 2024-04-18T10:06:55.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/4130882867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec072ec0 msgr2=0x7f9cec070c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:55.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 --2- 172.21.15.62:0/4130882867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec072ec0 0x7f9cec070c90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f9cd8009870 tx=0x7f9cd801c140 comp rx=0 tx=0).stop 2024-04-18T10:06:55.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/4130882867 shutdown_connections 2024-04-18T10:06:55.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 --2- 172.21.15.62:0/4130882867 >> v2:172.21.15.145:3300/0 conn(0x7f9cec0711d0 0x7f9cec148040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:55.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 --2- 172.21.15.62:0/4130882867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec072ec0 0x7f9cec070c90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 --2- 172.21.15.62:0/4130882867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cec067a70 0x7f9cec073370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/4130882867 >> 172.21.15.62:0/4130882867 conn(0x7f9cec06bca0 msgr2=0x7f9cec06c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/4130882867 shutdown_connections 2024-04-18T10:06:55.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/4130882867 wait complete. 2024-04-18T10:06:55.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 Processor -- start 2024-04-18T10:06:55.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 -- start start 2024-04-18T10:06:55.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec067a70 0x7f9cec0b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:55.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cec0711d0 0x7f9cec1e4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:55.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cec14dc60 0x7f9cec0af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:55.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec1456e0 con 0x7f9cec067a70 2024-04-18T10:06:55.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f9cec145560 con 0x7f9cec0711d0 2024-04-18T10:06:55.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf2bd8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f9cec145860 con 0x7f9cec14dc60 2024-04-18T10:06:55.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf0974640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec067a70 0x7f9cec0b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:55.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf0974640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec067a70 0x7f9cec0b34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:48990/0 (socket says 172.21.15.62:48990) 2024-04-18T10:06:55.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf0974640 1 -- 172.21.15.62:0/1293244136 learned_addr learned my addr 172.21.15.62:0/1293244136 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:55.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf1175640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cec14dc60 0x7f9cec0af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:55.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cebfff640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cec0711d0 0x7f9cec1e4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf0974640 1 -- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cec14dc60 msgr2=0x7f9cec0af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf0974640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cec14dc60 0x7f9cec0af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf0974640 1 -- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cec0711d0 msgr2=0x7f9cec1e4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf0974640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cec0711d0 0x7f9cec1e4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf0974640 1 -- 172.21.15.62:0/1293244136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cec1e4660 con 0x7f9cec067a70 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cf1175640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cec14dc60 0x7f9cec0af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.401+0000 7f9cebfff640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cec0711d0 0x7f9cec1e4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.405+0000 7f9cf0974640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec067a70 0x7f9cec0b34c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000d830 tx=0x7f9ce000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.405+0000 7f9ce97fa640 1 -- 172.21.15.62:0/1293244136 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce000b6c0 con 0x7f9cec067a70 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.405+0000 7f9ce97fa640 1 -- 172.21.15.62:0/1293244136 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ce0019070 con 0x7f9cec067a70 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.405+0000 7f9ce97fa640 1 -- 172.21.15.62:0/1293244136 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce000ba60 con 0x7f9cec067a70 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.405+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/1293244136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cec0b2a60 con 0x7f9cec067a70 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.405+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/1293244136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cec14f230 con 0x7f9cec067a70 2024-04-18T10:06:55.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.405+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/1293244136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cec1444b0 con 0x7f9cec067a70 2024-04-18T10:06:55.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.405+0000 7f9ce97fa640 1 -- 172.21.15.62:0/1293244136 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9ce001e460 con 0x7f9cec067a70 2024-04-18T10:06:55.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.405+0000 7f9ce97fa640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9cd0068380 0x7f9cd006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:55.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.405+0000 7f9ce97fa640 1 -- 172.21.15.62:0/1293244136 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5357+0+0 (secure 0 0 0) 0x7f9ce0086e90 con 0x7f9cec067a70 2024-04-18T10:06:55.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.405+0000 7f9cebfff640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9cd0068380 0x7f9cd006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:55.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.405+0000 7f9cebfff640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9cd0068380 0x7f9cd006a840 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9cd80ad000 tx=0x7f9cd8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:55.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.409+0000 7f9ce97fa640 1 -- 172.21.15.62:0/1293244136 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9cec1444b0 con 0x7f9cec067a70 2024-04-18T10:06:55.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:55.577+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/1293244136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9cec068ca0 con 0x7f9cec067a70 2024-04-18T10:06:56.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.265+0000 7f9ce97fa640 1 -- 172.21.15.62:0/1293244136 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v33) ==== 169+0+0 (secure 0 0 0) 0x7f9ce0057640 con 0x7f9cec067a70 2024-04-18T10:06:56.264 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'cephfs_data' removed 2024-04-18T10:06:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.269+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9cd0068380 msgr2=0x7f9cd006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.269+0000 7f9cf2bd8640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9cd0068380 0x7f9cd006a840 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9cd80ad000 tx=0x7f9cd8008040 comp rx=0 tx=0).stop 2024-04-18T10:06:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.269+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/1293244136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec067a70 msgr2=0x7f9cec0b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.269+0000 7f9cf2bd8640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec067a70 0x7f9cec0b34c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9ce000d830 tx=0x7f9ce000dcf0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.269+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/1293244136 shutdown_connections 2024-04-18T10:06:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.269+0000 7f9cf2bd8640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f9cd0068380 0x7f9cd006a840 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.269+0000 7f9cf2bd8640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9cec14dc60 0x7f9cec0af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.269+0000 7f9cf2bd8640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9cec0711d0 0x7f9cec1e4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.269+0000 7f9cf2bd8640 1 --2- 172.21.15.62:0/1293244136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9cec067a70 0x7f9cec0b34c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.269+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/1293244136 >> 172.21.15.62:0/1293244136 conn(0x7f9cec06bca0 msgr2=0x7f9cec13ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:56.269 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.269+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/1293244136 shutdown_connections 2024-04-18T10:06:56.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.269+0000 7f9cf2bd8640 1 -- 172.21.15.62:0/1293244136 wait complete. 2024-04-18T10:06:56.296 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-18T10:06:56.428 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55b0759640 1 Processor -- start 2024-04-18T10:06:56.429 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55b0759640 1 -- start start 2024-04-18T10:06:56.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55b0759640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f55a80b64b0 0x7f55a80b5220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:56.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55b0759640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f55a809f310 0x7f55a8149e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:56.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55b0759640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55a80b6890 con 0x7f55a80b5760 2024-04-18T10:06:56.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55b0759640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f55a809de30 con 0x7f55a809f310 2024-04-18T10:06:56.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55b0759640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f55a806c370 con 0x7f55a80b64b0 2024-04-18T10:06:56.430 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55aecf6640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f55a809f310 0x7f55a8149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:56.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55adcf4640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f55a80b5760 0x7f55a8149750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48728/0 (socket says 172.21.15.62:48728) 2024-04-18T10:06:56.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55adcf4640 1 -- 172.21.15.62:0/2601981006 learned_addr learned my addr 172.21.15.62:0/2601981006 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:56.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55ae4f5640 1 --2- 172.21.15.62:0/2601981006 >> v2:172.21.15.145:3300/0 conn(0x7f55a80b64b0 0x7f55a80b5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:56.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55aecf6640 1 -- 172.21.15.62:0/2601981006 >> v2:172.21.15.145:3300/0 conn(0x7f55a80b64b0 msgr2=0x7f55a80b5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:56.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55aecf6640 1 --2- 172.21.15.62:0/2601981006 >> v2:172.21.15.145:3300/0 conn(0x7f55a80b64b0 0x7f55a80b5220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55aecf6640 1 -- 172.21.15.62:0/2601981006 >> v1:172.21.15.62:6789/0 conn(0x7f55a80b5760 legacy=0x7f55a8149750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:56.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55ad4f3640 1 -- 172.21.15.62:0/2601981006 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3059265004 0 0) 0x7f55a80b6890 con 0x7f55a80b5760 2024-04-18T10:06:56.431 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.429+0000 7f55aecf6640 1 -- 172.21.15.62:0/2601981006 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a80b5e00 con 0x7f55a809f310 2024-04-18T10:06:56.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55aecf6640 1 --2- 172.21.15.62:0/2601981006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55a809f310 0x7f55a8149e60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f55a8068710 tx=0x7f55a001c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73509dab95fe149a server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:56.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55ad4f3640 1 -- 172.21.15.62:0/2601981006 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a00b2070 con 0x7f55a809f310 2024-04-18T10:06:56.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55ad4f3640 1 -- 172.21.15.62:0/2601981006 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a00a58a0 con 0x7f55a809f310 2024-04-18T10:06:56.432 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55ad4f3640 1 -- 172.21.15.62:0/2601981006 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a00a5c70 con 0x7f55a809f310 2024-04-18T10:06:56.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 -- 172.21.15.62:0/2601981006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55a809f310 msgr2=0x7f55a8149e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:56.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 --2- 172.21.15.62:0/2601981006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55a809f310 0x7f55a8149e60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f55a8068710 tx=0x7f55a001c7e0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 -- 172.21.15.62:0/2601981006 shutdown_connections 2024-04-18T10:06:56.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 --2- 172.21.15.62:0/2601981006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55a809f310 0x7f55a8149e60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 --2- 172.21.15.62:0/2601981006 >> v2:172.21.15.145:3300/0 conn(0x7f55a80b64b0 0x7f55a80b5220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.433 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 -- 172.21.15.62:0/2601981006 >> 172.21.15.62:0/2601981006 conn(0x7f55a80ba710 msgr2=0x7f55a80bab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:56.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 -- 172.21.15.62:0/2601981006 shutdown_connections 2024-04-18T10:06:56.434 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 -- 172.21.15.62:0/2601981006 wait complete. 2024-04-18T10:06:56.435 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 Processor -- start 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 -- start start 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55a809f310 0x7f55a80b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55a80b5760 0x7f55a81ddf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55a80b64b0 0x7f55a81de490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f55a8152e70 con 0x7f55a80b5760 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f55a809e5a0 con 0x7f55a80b64b0 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55b0759640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f55a8152ff0 con 0x7f55a809f310 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55adcf4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55a80b5760 0x7f55a81ddf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55adcf4640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55a80b5760 0x7f55a81ddf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49012/0 (socket says 172.21.15.62:49012) 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55adcf4640 1 -- 172.21.15.62:0/2228840065 learned_addr learned my addr 172.21.15.62:0/2228840065 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55aecf6640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55a80b64b0 0x7f55a81de490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55ae4f5640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55a809f310 0x7f55a80b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:56.436 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55adcf4640 1 -- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55a809f310 msgr2=0x7f55a80b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:56.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55adcf4640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55a809f310 0x7f55a80b3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55adcf4640 1 -- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55a80b64b0 msgr2=0x7f55a81de490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:56.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55adcf4640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55a80b64b0 0x7f55a81de490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55adcf4640 1 -- 172.21.15.62:0/2228840065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a80b1e60 con 0x7f55a80b5760 2024-04-18T10:06:56.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.433+0000 7f55adcf4640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55a80b5760 0x7f55a81ddf50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f559c00c520 tx=0x7f559c00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:56.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.437+0000 7f559affd640 1 -- 172.21.15.62:0/2228840065 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c018070 con 0x7f55a80b5760 2024-04-18T10:06:56.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.437+0000 7f559affd640 1 -- 172.21.15.62:0/2228840065 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f559c0074e0 con 0x7f55a80b5760 2024-04-18T10:06:56.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.437+0000 7f55b0759640 1 -- 172.21.15.62:0/2228840065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a80b35b0 con 0x7f55a80b5760 2024-04-18T10:06:56.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.437+0000 7f559affd640 1 -- 172.21.15.62:0/2228840065 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c0049f0 con 0x7f55a80b5760 2024-04-18T10:06:56.437 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.437+0000 7f55b0759640 1 -- 172.21.15.62:0/2228840065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a80b4820 con 0x7f55a80b5760 2024-04-18T10:06:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.437+0000 7f55b0759640 1 -- 172.21.15.62:0/2228840065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a8069230 con 0x7f55a80b5760 2024-04-18T10:06:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.437+0000 7f559affd640 1 -- 172.21.15.62:0/2228840065 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f559c01b070 con 0x7f55a80b5760 2024-04-18T10:06:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.437+0000 7f559affd640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f558c0683b0 0x7f558c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.437+0000 7f559affd640 1 -- 172.21.15.62:0/2228840065 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f559c085a60 con 0x7f55a80b5760 2024-04-18T10:06:56.438 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.437+0000 7f55ae4f5640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f558c0683b0 0x7f558c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:56.439 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.437+0000 7f55ae4f5640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f558c0683b0 0x7f558c06a870 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f55a400b930 tx=0x7f55a4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:56.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.441+0000 7f559affd640 1 -- 172.21.15.62:0/2228840065 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f559c0563b0 con 0x7f55a80b5760 2024-04-18T10:06:56.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.609+0000 7f55b0759640 1 -- 172.21.15.62:0/2228840065 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f55a8072030 con 0x7f55a80b5760 2024-04-18T10:06:56.613 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f559affd640 1 -- 172.21.15.62:0/2228840065 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f559c055cb0 con 0x7f55a80b5760 2024-04-18T10:06:56.615 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f55b0759640 1 -- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f558c0683b0 msgr2=0x7f558c06a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f55b0759640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f558c0683b0 0x7f558c06a870 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f55a400b930 tx=0x7f55a4008040 comp rx=0 tx=0).stop 2024-04-18T10:06:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f55b0759640 1 -- 172.21.15.62:0/2228840065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55a80b5760 msgr2=0x7f55a81ddf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f55b0759640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55a80b5760 0x7f55a81ddf50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f559c00c520 tx=0x7f559c00c9e0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f55b0759640 1 -- 172.21.15.62:0/2228840065 shutdown_connections 2024-04-18T10:06:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f55b0759640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f558c0683b0 0x7f558c06a870 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f55b0759640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55a80b64b0 0x7f55a81de490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f55b0759640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f55a80b5760 0x7f55a81ddf50 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f55b0759640 1 --2- 172.21.15.62:0/2228840065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55a809f310 0x7f55a80b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f55b0759640 1 -- 172.21.15.62:0/2228840065 >> 172.21.15.62:0/2228840065 conn(0x7f55a80ba710 msgr2=0x7f55a806eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f55b0759640 1 -- 172.21.15.62:0/2228840065 shutdown_connections 2024-04-18T10:06:56.616 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.613+0000 7f55b0759640 1 -- 172.21.15.62:0/2228840065 wait complete. 2024-04-18T10:06:56.633 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args' 2024-04-18T10:06:56.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb0805e8640 1 Processor -- start 2024-04-18T10:06:56.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb0805e8640 1 -- start start 2024-04-18T10:06:56.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb0805e8640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb0780af4f0 0x7fb0780b1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:56.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb0805e8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0780b65e0 0x7fb07806bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:56.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb0805e8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07806bad0 con 0x7fb0780af8d0 2024-04-18T10:06:56.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb0805e8640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fb078147db0 con 0x7fb0780af4f0 2024-04-18T10:06:56.761 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb0805e8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fb0780b6b90 con 0x7fb0780b65e0 2024-04-18T10:06:56.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb07e384640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb0780af4f0 0x7fb0780b1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:56.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb07eb85640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0780b65e0 0x7fb07806bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:56.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb07db83640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fb0780af8d0 0x7fb078148550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.62:52198/0 (socket says 172.21.15.62:52198) 2024-04-18T10:06:56.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb07db83640 1 -- 172.21.15.62:0/112460807 learned_addr learned my addr 172.21.15.62:0/112460807 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:56.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb07e384640 1 -- 172.21.15.62:0/112460807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0780b65e0 msgr2=0x7fb07806bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:56.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb07e384640 1 --2- 172.21.15.62:0/112460807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0780b65e0 0x7fb07806bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.762 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb07e384640 1 -- 172.21.15.62:0/112460807 >> v1:172.21.15.145:6789/0 conn(0x7fb0780af8d0 legacy=0x7fb078148550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:56.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb07e384640 1 -- 172.21.15.62:0/112460807 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb078148030 con 0x7fb0780af4f0 2024-04-18T10:06:56.763 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.761+0000 7fb07eb85640 1 --2- 172.21.15.62:0/112460807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0780b65e0 0x7fb07806bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-18T10:06:56.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb07e384640 1 --2- 172.21.15.62:0/112460807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0780af4f0 0x7fb0780b1120 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb0740097c0 tx=0x7fb07401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db31f9f5ef5e9470 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:56.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb07d382640 1 -- 172.21.15.62:0/112460807 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0740b1070 con 0x7fb0780af4f0 2024-04-18T10:06:56.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb07d382640 1 -- 172.21.15.62:0/112460807 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0740ac410 con 0x7fb0780af4f0 2024-04-18T10:06:56.765 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb07d382640 1 -- 172.21.15.62:0/112460807 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0740ac7e0 con 0x7fb0780af4f0 2024-04-18T10:06:56.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb0805e8640 1 -- 172.21.15.62:0/112460807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0780af4f0 msgr2=0x7fb0780b1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:56.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb0805e8640 1 --2- 172.21.15.62:0/112460807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0780af4f0 0x7fb0780b1120 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb0740097c0 tx=0x7fb07401c4a0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb0805e8640 1 -- 172.21.15.62:0/112460807 shutdown_connections 2024-04-18T10:06:56.766 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb0805e8640 1 --2- 172.21.15.62:0/112460807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0780b65e0 0x7fb07806bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb0805e8640 1 --2- 172.21.15.62:0/112460807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0780af4f0 0x7fb0780b1120 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb0805e8640 1 -- 172.21.15.62:0/112460807 >> 172.21.15.62:0/112460807 conn(0x7fb0780b6fd0 msgr2=0x7fb0780b73e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:56.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb0805e8640 1 -- 172.21.15.62:0/112460807 shutdown_connections 2024-04-18T10:06:56.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb0805e8640 1 -- 172.21.15.62:0/112460807 wait complete. 2024-04-18T10:06:56.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb0805e8640 1 Processor -- start 2024-04-18T10:06:56.767 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.765+0000 7fb0805e8640 1 -- start start 2024-04-18T10:06:56.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb0805e8640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0780af4f0 0x7fb078149170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:56.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb0805e8640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0780af8d0 0x7fb0781e0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:56.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb0805e8640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0780b65e0 0x7fb07809f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:56.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb0805e8640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb078154ff0 con 0x7fb0780b65e0 2024-04-18T10:06:56.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb0805e8640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fb078154e70 con 0x7fb0780af8d0 2024-04-18T10:06:56.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb0805e8640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fb078155170 con 0x7fb0780af4f0 2024-04-18T10:06:56.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb07eb85640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0780b65e0 0x7fb07809f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:56.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb07eb85640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0780b65e0 0x7fb07809f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49026/0 (socket says 172.21.15.62:49026) 2024-04-18T10:06:56.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb07eb85640 1 -- 172.21.15.62:0/3284623136 learned_addr learned my addr 172.21.15.62:0/3284623136 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:56.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb07e384640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0780af4f0 0x7fb078149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:56.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb07db83640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0780af8d0 0x7fb0781e0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:56.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb07eb85640 1 -- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0780af4f0 msgr2=0x7fb078149170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:56.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb07eb85640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0780af4f0 0x7fb078149170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb07eb85640 1 -- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0780af8d0 msgr2=0x7fb0781e0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:56.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb07eb85640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0780af8d0 0x7fb0781e0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:56.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb07eb85640 1 -- 172.21.15.62:0/3284623136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07814a1f0 con 0x7fb0780b65e0 2024-04-18T10:06:56.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb07e384640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0780af4f0 0x7fb078149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:56.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb07eb85640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0780b65e0 0x7fb07809f5c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb07000efc0 tx=0x7fb070009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:56.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb06effd640 1 -- 172.21.15.62:0/3284623136 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb070018030 con 0x7fb0780b65e0 2024-04-18T10:06:56.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb06effd640 1 -- 172.21.15.62:0/3284623136 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb07000cde0 con 0x7fb0780b65e0 2024-04-18T10:06:56.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb0805e8640 1 -- 172.21.15.62:0/3284623136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0780a0370 con 0x7fb0780b65e0 2024-04-18T10:06:56.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb06effd640 1 -- 172.21.15.62:0/3284623136 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb070003bc0 con 0x7fb0780b65e0 2024-04-18T10:06:56.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb0805e8640 1 -- 172.21.15.62:0/3284623136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0780b42d0 con 0x7fb0780b65e0 2024-04-18T10:06:56.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb0805e8640 1 -- 172.21.15.62:0/3284623136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb078068b90 con 0x7fb0780b65e0 2024-04-18T10:06:56.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb06effd640 1 -- 172.21.15.62:0/3284623136 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb07001b070 con 0x7fb0780b65e0 2024-04-18T10:06:56.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb06effd640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb05c068380 0x7fb05c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:56.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.769+0000 7fb06effd640 1 -- 172.21.15.62:0/3284623136 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fb070087060 con 0x7fb0780b65e0 2024-04-18T10:06:56.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.773+0000 7fb07e384640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb05c068380 0x7fb05c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:56.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.773+0000 7fb07e384640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb05c068380 0x7fb05c06a840 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb0740ae000 tx=0x7fb074008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:56.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.777+0000 7fb06effd640 1 -- 172.21.15.62:0/3284623136 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb070022070 con 0x7fb0780b65e0 2024-04-18T10:06:56.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:56.997+0000 7fb0805e8640 1 -- 172.21.15.62:0/3284623136 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args"]} v 0) -- 0x7fb07814a3c0 con 0x7fb0780b65e0 2024-04-18T10:06:57.267 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.265+0000 7fb06effd640 1 -- 172.21.15.62:0/3284623136 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_cephfs_shell.TestCD.test_cd_with_args"]}]=0 v79) ==== 134+0+0 (secure 0 0 0) 0x7fb070010040 con 0x7fb0780b65e0 2024-04-18T10:06:57.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.269+0000 7fb0805e8640 1 -- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb05c068380 msgr2=0x7fb05c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:57.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.269+0000 7fb0805e8640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb05c068380 0x7fb05c06a840 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb0740ae000 tx=0x7fb074008040 comp rx=0 tx=0).stop 2024-04-18T10:06:57.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.269+0000 7fb0805e8640 1 -- 172.21.15.62:0/3284623136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0780b65e0 msgr2=0x7fb07809f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:57.270 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.269+0000 7fb0805e8640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0780b65e0 0x7fb07809f5c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb07000efc0 tx=0x7fb070009f90 comp rx=0 tx=0).stop 2024-04-18T10:06:57.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.269+0000 7fb0805e8640 1 -- 172.21.15.62:0/3284623136 shutdown_connections 2024-04-18T10:06:57.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.269+0000 7fb0805e8640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb05c068380 0x7fb05c06a840 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:57.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.269+0000 7fb0805e8640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb0780b65e0 0x7fb07809f5c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:57.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.269+0000 7fb0805e8640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0780af8d0 0x7fb0781e0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:57.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.269+0000 7fb0805e8640 1 --2- 172.21.15.62:0/3284623136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0780af4f0 0x7fb078149170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:57.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.269+0000 7fb0805e8640 1 -- 172.21.15.62:0/3284623136 >> 172.21.15.62:0/3284623136 conn(0x7fb0780b6fd0 msgr2=0x7fb078148550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:57.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.269+0000 7fb0805e8640 1 -- 172.21.15.62:0/3284623136 shutdown_connections 2024-04-18T10:06:57.271 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:57.269+0000 7fb0805e8640 1 -- 172.21.15.62:0/3284623136 wait complete. 2024-04-18T10:06:57.294 INFO:tasks.cephfs_test_runner:Test that when cd is issued with an argument, CWD is changed ... ERROR 2024-04-18T10:06:57.294 INFO:tasks.cephfs_test_runner: 2024-04-18T10:06:57.294 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-18T10:06:57.294 INFO:tasks.cephfs_test_runner:ERROR: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-04-18T10:06:57.294 INFO:tasks.cephfs_test_runner:Test that when cd is issued with an argument, CWD is changed 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_08d35a8d8529783882dd092c73c0b27be41c4d86/qa/tasks/cephfs/test_cephfs_shell.py", line 602, in test_cd_with_args 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: output = self.get_cephfs_shell_script_output(script) 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_08d35a8d8529783882dd092c73c0b27be41c4d86/qa/tasks/cephfs/test_cephfs_shell.py", line 161, in get_cephfs_shell_script_output 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: return ensure_str(self.run_cephfs_shell_script( 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_08d35a8d8529783882dd092c73c0b27be41c4d86/qa/tasks/cephfs/test_cephfs_shell.py", line 153, in run_cephfs_shell_script 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: return mount_x.client_remote.run(args=args, stdout=stdout, 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: r.wait() 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-04-18T10:06:57.295 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 127: 'cephfs-shell -b /tmp/test-cephfs86v2e119' 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner: 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner:Ran 1 test in 37.678s 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner: 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner: 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner:ERROR: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner:Test that when cd is issued with an argument, CWD is changed 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_08d35a8d8529783882dd092c73c0b27be41c4d86/qa/tasks/cephfs/test_cephfs_shell.py", line 602, in test_cd_with_args 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner: output = self.get_cephfs_shell_script_output(script) 2024-04-18T10:06:57.296 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_08d35a8d8529783882dd092c73c0b27be41c4d86/qa/tasks/cephfs/test_cephfs_shell.py", line 161, in get_cephfs_shell_script_output 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner: return ensure_str(self.run_cephfs_shell_script( 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph_08d35a8d8529783882dd092c73c0b27be41c4d86/qa/tasks/cephfs/test_cephfs_shell.py", line 153, in run_cephfs_shell_script 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner: return mount_x.client_remote.run(args=args, stdout=stdout, 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner: r.wait() 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi062 with status 127: 'cephfs-shell -b /tmp/test-cephfs86v2e119' 2024-04-18T10:06:57.297 INFO:tasks.cephfs_test_runner: 2024-04-18T10:06:57.298 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_08d35a8d8529783882dd092c73c0b27be41c4d86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-04-18T10:06:57.535 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1266d81a082e46e284e8cea1bbb9f61d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_08d35a8d8529783882dd092c73c0b27be41c4d86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD) 2024-04-18T10:06:57.536 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-18T10:06:57.545 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-18T10:06:57.573 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-18T10:06:57.573 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-04-18T10:06:57.573 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-18T10:06:57.586 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16336528k,nr_inodes=4084132,mode=755,inode64 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274692k,mode=755,inode64 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15920 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274688k,nr_inodes=818672,mode=700,uid=1000,gid=1271,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274692k,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 2024-04-18T10:06:57.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:57.586 DEBUG:teuthology.orchestra.run.smithi062:> ip netns list 2024-04-18T10:06:57.637 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-18T10:06:57.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:57.638 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-18T10:06:57.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-18T10:06:57.695 DEBUG:teuthology.orchestra.run.smithi062:> sudo ip link delete ceph-brx 2024-04-18T10:06:57.878 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-18T10:06:57.887 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-18T10:06:57.887 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T10:06:58.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.017+0000 7f28583ee640 1 Processor -- start 2024-04-18T10:06:58.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.017+0000 7f28583ee640 1 -- start start 2024-04-18T10:06:58.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f28583ee640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f28500b9d90 0x7f285006d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f28583ee640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2850145f20 0x7f28501504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f28583ee640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28500ba170 con 0x7f285009be30 2024-04-18T10:06:58.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f28583ee640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f285009cd30 con 0x7f2850145f20 2024-04-18T10:06:58.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f28583ee640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f285014a060 con 0x7f28500b9d90 2024-04-18T10:06:58.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f285698b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2850145f20 0x7f28501504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f2855989640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f285009be30 0x7f285006d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48736/0 (socket says 172.21.15.62:48736) 2024-04-18T10:06:58.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f2855989640 1 -- 172.21.15.62:0/2863546643 learned_addr learned my addr 172.21.15.62:0/2863546643 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:58.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f285618a640 1 --2- 172.21.15.62:0/2863546643 >> v2:172.21.15.145:3300/0 conn(0x7f28500b9d90 0x7f285006d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f285698b640 1 -- 172.21.15.62:0/2863546643 >> v2:172.21.15.145:3300/0 conn(0x7f28500b9d90 msgr2=0x7f285006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:58.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f285698b640 1 --2- 172.21.15.62:0/2863546643 >> v2:172.21.15.145:3300/0 conn(0x7f28500b9d90 0x7f285006d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f285698b640 1 -- 172.21.15.62:0/2863546643 >> v1:172.21.15.62:6789/0 conn(0x7f285009be30 legacy=0x7f285006d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f285698b640 1 -- 172.21.15.62:0/2863546643 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2850149860 con 0x7f2850145f20 2024-04-18T10:06:58.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f2855188640 1 -- 172.21.15.62:0/2863546643 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3569780731 0 0) 0x7f28500ba170 con 0x7f285009be30 2024-04-18T10:06:58.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f285698b640 1 --2- 172.21.15.62:0/2863546643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2850145f20 0x7f28501504e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f285006c9d0 tx=0x7f284001c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc8bf6ae1fdd354 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:58.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.021+0000 7f2855188640 1 -- 172.21.15.62:0/2863546643 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28400b2070 con 0x7f2850145f20 2024-04-18T10:06:58.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f2855188640 1 -- 172.21.15.62:0/2863546643 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28400a57a0 con 0x7f2850145f20 2024-04-18T10:06:58.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f2855188640 1 -- 172.21.15.62:0/2863546643 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28400a5b70 con 0x7f2850145f20 2024-04-18T10:06:58.025 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 -- 172.21.15.62:0/2863546643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2850145f20 msgr2=0x7f28501504e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 --2- 172.21.15.62:0/2863546643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2850145f20 0x7f28501504e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f285006c9d0 tx=0x7f284001c810 comp rx=0 tx=0).stop 2024-04-18T10:06:58.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 -- 172.21.15.62:0/2863546643 shutdown_connections 2024-04-18T10:06:58.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 --2- 172.21.15.62:0/2863546643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2850145f20 0x7f28501504e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 --2- 172.21.15.62:0/2863546643 >> v2:172.21.15.145:3300/0 conn(0x7f28500b9d90 0x7f285006d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 -- 172.21.15.62:0/2863546643 >> 172.21.15.62:0/2863546643 conn(0x7f285006c0d0 msgr2=0x7f28500b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:58.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 -- 172.21.15.62:0/2863546643 shutdown_connections 2024-04-18T10:06:58.026 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 -- 172.21.15.62:0/2863546643 wait complete. 2024-04-18T10:06:58.029 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 Processor -- start 2024-04-18T10:06:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 -- start start 2024-04-18T10:06:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f285009be30 0x7f2850148420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28500b9d90 0x7f285013c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2850145f20 0x7f285013c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f28500b3fa0 con 0x7f2850145f20 2024-04-18T10:06:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f28500b3e20 con 0x7f28500b9d90 2024-04-18T10:06:58.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f285006de60 con 0x7f285009be30 2024-04-18T10:06:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f285698b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2850145f20 0x7f285013c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f285698b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2850145f20 0x7f285013c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49044/0 (socket says 172.21.15.62:49044) 2024-04-18T10:06:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f285698b640 1 -- 172.21.15.62:0/3254046623 learned_addr learned my addr 172.21.15.62:0/3254046623 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f285618a640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f285009be30 0x7f2850148420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f2855989640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28500b9d90 0x7f285013c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f285698b640 1 -- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f285009be30 msgr2=0x7f2850148420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f285698b640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f285009be30 0x7f2850148420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f285698b640 1 -- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28500b9d90 msgr2=0x7f285013c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f285698b640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28500b9d90 0x7f285013c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f285698b640 1 -- 172.21.15.62:0/3254046623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28501479c0 con 0x7f2850145f20 2024-04-18T10:06:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f285618a640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f285009be30 0x7f2850148420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f285698b640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2850145f20 0x7f285013c7d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2840019420 tx=0x7f28400195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f284effd640 1 -- 172.21.15.62:0/3254046623 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28400bd020 con 0x7f2850145f20 2024-04-18T10:06:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f284effd640 1 -- 172.21.15.62:0/3254046623 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28400b3040 con 0x7f2850145f20 2024-04-18T10:06:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 -- 172.21.15.62:0/3254046623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28501dea90 con 0x7f2850145f20 2024-04-18T10:06:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f284effd640 1 -- 172.21.15.62:0/3254046623 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28400b2050 con 0x7f2850145f20 2024-04-18T10:06:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.025+0000 7f28583ee640 1 -- 172.21.15.62:0/3254046623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2850147ca0 con 0x7f2850145f20 2024-04-18T10:06:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.029+0000 7f28583ee640 1 -- 172.21.15.62:0/3254046623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2850068b90 con 0x7f2850145f20 2024-04-18T10:06:58.032 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.029+0000 7f284effd640 1 -- 172.21.15.62:0/3254046623 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2840003ba0 con 0x7f2850145f20 2024-04-18T10:06:58.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.029+0000 7f284effd640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f283c068380 0x7f283c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.029+0000 7f284effd640 1 -- 172.21.15.62:0/3254046623 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f2840121050 con 0x7f2850145f20 2024-04-18T10:06:58.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.029+0000 7f285618a640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f283c068380 0x7f283c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.029+0000 7f285618a640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f283c068380 0x7f283c06a840 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2850073aa0 tx=0x7f2848008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:58.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.033+0000 7f284effd640 1 -- 172.21.15.62:0/3254046623 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f28400c7020 con 0x7f2850145f20 2024-04-18T10:06:58.193 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.193+0000 7f28583ee640 1 -- 172.21.15.62:0/3254046623 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f28500ba170 con 0x7f283c068380 2024-04-18T10:06:58.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.193+0000 7f284effd640 1 -- 172.21.15.62:0/3254046623 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13485 (secure 0 0 0) 0x7f28500ba170 con 0x7f283c068380 2024-04-18T10:06:58.195 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:58.196 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-18T10:06:58.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.197+0000 7f28583ee640 1 -- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f283c068380 msgr2=0x7f283c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.197+0000 7f28583ee640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f283c068380 0x7f283c06a840 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2850073aa0 tx=0x7f2848008040 comp rx=0 tx=0).stop 2024-04-18T10:06:58.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.197+0000 7f28583ee640 1 -- 172.21.15.62:0/3254046623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2850145f20 msgr2=0x7f285013c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.197+0000 7f28583ee640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2850145f20 0x7f285013c7d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2840019420 tx=0x7f28400195b0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.197+0000 7f28583ee640 1 -- 172.21.15.62:0/3254046623 shutdown_connections 2024-04-18T10:06:58.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.197+0000 7f28583ee640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f283c068380 0x7f283c06a840 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.197+0000 7f28583ee640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2850145f20 0x7f285013c7d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.197+0000 7f28583ee640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28500b9d90 0x7f285013c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.197+0000 7f28583ee640 1 --2- 172.21.15.62:0/3254046623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f285009be30 0x7f2850148420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.197+0000 7f28583ee640 1 -- 172.21.15.62:0/3254046623 >> 172.21.15.62:0/3254046623 conn(0x7f285006c0d0 msgr2=0x7f285006d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:58.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.197+0000 7f28583ee640 1 -- 172.21.15.62:0/3254046623 shutdown_connections 2024-04-18T10:06:58.199 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.197+0000 7f28583ee640 1 -- 172.21.15.62:0/3254046623 wait complete. 2024-04-18T10:06:58.215 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-04-18T10:06:57.144780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":220188,"kb_used_data":4024,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754532,"statfs":{"total":773094113280,"available":772868640768,"internally_reserved":0,"allocated":4120576,"data_stored":2312437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2851,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-46,"num_write_kb":-15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002027"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":192,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-18T10:06:56.300560+0000","last_change":"2024-04-18T10:05:49.183850+0000","last_active":"2024-04-18T10:06:56.300560+0000","last_peered":"2024-04-18T10:06:56.300560+0000","last_clean":"2024-04-18T10:06:56.300560+0000","last_became_active":"2024-04-18T10:05:49.181471+0000","last_became_peered":"2024-04-18T10:05:49.181471+0000","last_unstale":"2024-04-18T10:06:56.300560+0000","last_undegraded":"2024-04-18T10:06:56.300560+0000","last_fullsized":"2024-04-18T10:06:56.300560+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_clean_scrub_stamp":"2024-04-18T10:05:47.511486+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:06:51.214547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":731135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":958464,"data_stored":731135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T10:06:58.216 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T10:06:58.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd85ada0640 1 Processor -- start 2024-04-18T10:06:58.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd85ada0640 1 -- start start 2024-04-18T10:06:58.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd85ada0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd8540b6260 0x7fd854148260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd85ada0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8540b7850 con 0x7fd8541487a0 2024-04-18T10:06:58.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd85ada0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8540b4c20 con 0x7fd8540b6640 2024-04-18T10:06:58.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd85ada0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fd854148cb0 con 0x7fd8540b6260 2024-04-18T10:06:58.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd858b3c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd8540b6260 0x7fd854148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd858b3c640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd8540b6260 0x7fd854148260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49050/0 (socket says 172.21.15.62:49050) 2024-04-18T10:06:58.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd858b3c640 1 -- 172.21.15.62:0/2898884626 learned_addr learned my addr 172.21.15.62:0/2898884626 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:58.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd853fff640 1 --1- 172.21.15.62:0/2898884626 >> v1:172.21.15.62:6789/0 conn(0x7fd8541487a0 0x7fd8541419a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48750/0 (socket says 172.21.15.62:48750) 2024-04-18T10:06:58.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd858b3c640 1 -- 172.21.15.62:0/2898884626 >> v1:172.21.15.145:6789/0 conn(0x7fd8540b6640 legacy=0x7fd8540b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:58.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd858b3c640 1 -- 172.21.15.62:0/2898884626 >> v1:172.21.15.62:6789/0 conn(0x7fd8541487a0 legacy=0x7fd8541419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd858b3c640 1 -- 172.21.15.62:0/2898884626 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8541420b0 con 0x7fd8540b6260 2024-04-18T10:06:58.339 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.337+0000 7fd858b3c640 1 --2- 172.21.15.62:0/2898884626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8540b6260 0x7fd854148260 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd848009870 tx=0x7fd84801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a43c816c737fa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:58.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd8537fe640 1 -- 172.21.15.62:0/2898884626 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8480b0070 con 0x7fd8540b6260 2024-04-18T10:06:58.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd8537fe640 1 -- 172.21.15.62:0/2898884626 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8480ab600 con 0x7fd8540b6260 2024-04-18T10:06:58.340 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd8537fe640 1 -- 172.21.15.62:0/2898884626 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8480ab9d0 con 0x7fd8540b6260 2024-04-18T10:06:58.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 -- 172.21.15.62:0/2898884626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8540b6260 msgr2=0x7fd854148260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 --2- 172.21.15.62:0/2898884626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8540b6260 0x7fd854148260 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd848009870 tx=0x7fd84801c400 comp rx=0 tx=0).stop 2024-04-18T10:06:58.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 -- 172.21.15.62:0/2898884626 shutdown_connections 2024-04-18T10:06:58.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 --2- 172.21.15.62:0/2898884626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8540b6260 0x7fd854148260 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 -- 172.21.15.62:0/2898884626 >> 172.21.15.62:0/2898884626 conn(0x7fd8540bd5c0 msgr2=0x7fd8540bd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:58.341 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 -- 172.21.15.62:0/2898884626 shutdown_connections 2024-04-18T10:06:58.342 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 -- 172.21.15.62:0/2898884626 wait complete. 2024-04-18T10:06:58.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 Processor -- start 2024-04-18T10:06:58.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 -- start start 2024-04-18T10:06:58.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8540b6260 0x7fd854130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8540b6640 0x7fd854145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8541487a0 0x7fd8541455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd854156250 con 0x7fd8541487a0 2024-04-18T10:06:58.344 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fd8541560d0 con 0x7fd8540b6640 2024-04-18T10:06:58.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd8541563d0 con 0x7fd8540b6260 2024-04-18T10:06:58.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85933d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8541487a0 0x7fd8541455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85933d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8541487a0 0x7fd8541455c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49060/0 (socket says 172.21.15.62:49060) 2024-04-18T10:06:58.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85933d640 1 -- 172.21.15.62:0/550794958 learned_addr learned my addr 172.21.15.62:0/550794958 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:58.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd858b3c640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8540b6260 0x7fd854130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd853fff640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8540b6640 0x7fd854145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85933d640 1 -- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8540b6260 msgr2=0x7fd854130c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85933d640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8540b6260 0x7fd854130c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.345 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85933d640 1 -- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8540b6640 msgr2=0x7fd854145080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85933d640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8540b6640 0x7fd854145080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85933d640 1 -- 172.21.15.62:0/550794958 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8541444a0 con 0x7fd8541487a0 2024-04-18T10:06:58.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd853fff640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8540b6640 0x7fd854145080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:58.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85933d640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8541487a0 0x7fd8541455c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd84c00c780 tx=0x7fd84c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:58.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd8517fa640 1 -- 172.21.15.62:0/550794958 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84c016020 con 0x7fd8541487a0 2024-04-18T10:06:58.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd8517fa640 1 -- 172.21.15.62:0/550794958 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd84c00a9f0 con 0x7fd8541487a0 2024-04-18T10:06:58.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 -- 172.21.15.62:0/550794958 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd854143170 con 0x7fd8541487a0 2024-04-18T10:06:58.346 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd8517fa640 1 -- 172.21.15.62:0/550794958 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84c00ae20 con 0x7fd8541487a0 2024-04-18T10:06:58.347 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.341+0000 7fd85ada0640 1 -- 172.21.15.62:0/550794958 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd854148260 con 0x7fd8541487a0 2024-04-18T10:06:58.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.345+0000 7fd85ada0640 1 -- 172.21.15.62:0/550794958 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd854068b90 con 0x7fd8541487a0 2024-04-18T10:06:58.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.345+0000 7fd8517fa640 1 -- 172.21.15.62:0/550794958 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd84c019020 con 0x7fd8541487a0 2024-04-18T10:06:58.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.345+0000 7fd8517fa640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd838068380 0x7fd83806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.345+0000 7fd8517fa640 1 -- 172.21.15.62:0/550794958 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fd84c086520 con 0x7fd8541487a0 2024-04-18T10:06:58.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.345+0000 7fd858b3c640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd838068380 0x7fd83806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.348 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.345+0000 7fd858b3c640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd838068380 0x7fd83806a840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd8480ad000 tx=0x7fd8480a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:58.350 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.349+0000 7fd8517fa640 1 -- 172.21.15.62:0/550794958 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd84c020030 con 0x7fd8541487a0 2024-04-18T10:06:58.510 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.509+0000 7fd85ada0640 1 -- 172.21.15.62:0/550794958 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd85406bad0 con 0x7fd838068380 2024-04-18T10:06:58.512 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd8517fa640 1 -- 172.21.15.62:0/550794958 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13485 (secure 0 0 0) 0x7fd85406bad0 con 0x7fd838068380 2024-04-18T10:06:58.512 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:58.513 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-18T10:06:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd85ada0640 1 -- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd838068380 msgr2=0x7fd83806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd85ada0640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd838068380 0x7fd83806a840 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd8480ad000 tx=0x7fd8480a6000 comp rx=0 tx=0).stop 2024-04-18T10:06:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd85ada0640 1 -- 172.21.15.62:0/550794958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8541487a0 msgr2=0x7fd8541455c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd85ada0640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8541487a0 0x7fd8541455c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd84c00c780 tx=0x7fd84c00cc40 comp rx=0 tx=0).stop 2024-04-18T10:06:58.514 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd85ada0640 1 -- 172.21.15.62:0/550794958 shutdown_connections 2024-04-18T10:06:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd85ada0640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd838068380 0x7fd83806a840 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd85ada0640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd8541487a0 0x7fd8541455c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd85ada0640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8540b6640 0x7fd854145080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd85ada0640 1 --2- 172.21.15.62:0/550794958 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8540b6260 0x7fd854130c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd85ada0640 1 -- 172.21.15.62:0/550794958 >> 172.21.15.62:0/550794958 conn(0x7fd8540bd5c0 msgr2=0x7fd8540b7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd85ada0640 1 -- 172.21.15.62:0/550794958 shutdown_connections 2024-04-18T10:06:58.515 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.513+0000 7fd85ada0640 1 -- 172.21.15.62:0/550794958 wait complete. 2024-04-18T10:06:58.531 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-04-18T10:06:57.144780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":220188,"kb_used_data":4024,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754532,"statfs":{"total":773094113280,"available":772868640768,"internally_reserved":0,"allocated":4120576,"data_stored":2312437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2851,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-46,"num_write_kb":-15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002027"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":192,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-18T10:06:56.300560+0000","last_change":"2024-04-18T10:05:49.183850+0000","last_active":"2024-04-18T10:06:56.300560+0000","last_peered":"2024-04-18T10:06:56.300560+0000","last_clean":"2024-04-18T10:06:56.300560+0000","last_became_active":"2024-04-18T10:05:49.181471+0000","last_became_peered":"2024-04-18T10:05:49.181471+0000","last_unstale":"2024-04-18T10:06:56.300560+0000","last_undegraded":"2024-04-18T10:06:56.300560+0000","last_fullsized":"2024-04-18T10:06:56.300560+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_clean_scrub_stamp":"2024-04-18T10:05:47.511486+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:06:51.214547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":731135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":958464,"data_stored":731135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T10:06:58.532 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-18T10:06:58.532 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T10:06:58.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe98db58640 1 Processor -- start 2024-04-18T10:06:58.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe98db58640 1 -- start start 2024-04-18T10:06:58.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe98db58640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9880b9d90 0x7fe98806d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe98db58640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe98809be30 0x7fe98806d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe98db58640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9880ba170 con 0x7fe98806dc90 2024-04-18T10:06:58.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe98db58640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fe98809cd30 con 0x7fe98809be30 2024-04-18T10:06:58.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe98db58640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fe98814a060 con 0x7fe9880b9d90 2024-04-18T10:06:58.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe987fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe98806dc90 0x7fe988144f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48760/0 (socket says 172.21.15.62:48760) 2024-04-18T10:06:58.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe987fff640 1 -- 172.21.15.62:0/87473154 learned_addr learned my addr 172.21.15.62:0/87473154 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:06:58.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe986ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fe98809be30 0x7fe98806d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe9877fe640 1 --2- 172.21.15.62:0/87473154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9880b9d90 0x7fe98806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe9867fc640 1 -- 172.21.15.62:0/87473154 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2546117195 0 0) 0x7fe9880ba170 con 0x7fe98806dc90 2024-04-18T10:06:58.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe986ffd640 1 -- 172.21.15.62:0/87473154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9880b9d90 msgr2=0x7fe98806d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:58.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe986ffd640 1 --2- 172.21.15.62:0/87473154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9880b9d90 0x7fe98806d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe986ffd640 1 -- 172.21.15.62:0/87473154 >> v1:172.21.15.62:6789/0 conn(0x7fe98806dc90 legacy=0x7fe988144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.649+0000 7fe986ffd640 1 -- 172.21.15.62:0/87473154 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe988149860 con 0x7fe98809be30 2024-04-18T10:06:58.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe986ffd640 1 --2- 172.21.15.62:0/87473154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98809be30 0x7fe98806d750 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe97c0097c0 tx=0x7fe97c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6262e8660d850ca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:58.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe9867fc640 1 -- 172.21.15.62:0/87473154 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0b0070 con 0x7fe98809be30 2024-04-18T10:06:58.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe9867fc640 1 -- 172.21.15.62:0/87473154 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe97c0ab600 con 0x7fe98809be30 2024-04-18T10:06:58.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe9867fc640 1 -- 172.21.15.62:0/87473154 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0ab9d0 con 0x7fe98809be30 2024-04-18T10:06:58.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 -- 172.21.15.62:0/87473154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98809be30 msgr2=0x7fe98806d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 --2- 172.21.15.62:0/87473154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98809be30 0x7fe98806d750 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe97c0097c0 tx=0x7fe97c01c380 comp rx=0 tx=0).stop 2024-04-18T10:06:58.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 -- 172.21.15.62:0/87473154 shutdown_connections 2024-04-18T10:06:58.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 --2- 172.21.15.62:0/87473154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe98809be30 0x7fe98806d750 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 --2- 172.21.15.62:0/87473154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9880b9d90 0x7fe98806d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 -- 172.21.15.62:0/87473154 >> 172.21.15.62:0/87473154 conn(0x7fe98806c0d0 msgr2=0x7fe9880b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:58.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 -- 172.21.15.62:0/87473154 shutdown_connections 2024-04-18T10:06:58.654 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 -- 172.21.15.62:0/87473154 wait complete. 2024-04-18T10:06:58.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 Processor -- start 2024-04-18T10:06:58.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 -- start start 2024-04-18T10:06:58.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98806dc90 0x7fe988072d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe98809be30 0x7fe9880727e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9880b9d90 0x7fe9880720a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fe988145fb0 con 0x7fe9880b9d90 2024-04-18T10:06:58.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fe988145e30 con 0x7fe98809be30 2024-04-18T10:06:58.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe98db58640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fe988146130 con 0x7fe98806dc90 2024-04-18T10:06:58.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe987fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9880b9d90 0x7fe9880720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe987fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9880b9d90 0x7fe9880720a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49070/0 (socket says 172.21.15.62:49070) 2024-04-18T10:06:58.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe987fff640 1 -- 172.21.15.62:0/4104137311 learned_addr learned my addr 172.21.15.62:0/4104137311 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:58.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe9877fe640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98806dc90 0x7fe988072d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe986ffd640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe98809be30 0x7fe9880727e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe987fff640 1 -- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98806dc90 msgr2=0x7fe988072d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe987fff640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98806dc90 0x7fe988072d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.658 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe987fff640 1 -- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe98809be30 msgr2=0x7fe9880727e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe987fff640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe98809be30 0x7fe9880727e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe987fff640 1 -- 172.21.15.62:0/4104137311 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9880b08c0 con 0x7fe9880b9d90 2024-04-18T10:06:58.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe986ffd640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe98809be30 0x7fe9880727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:06:58.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe987fff640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9880b9d90 0x7fe9880720a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe980001570 tx=0x7fe98000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:58.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.653+0000 7fe96ffff640 1 -- 172.21.15.62:0/4104137311 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe980018070 con 0x7fe9880b9d90 2024-04-18T10:06:58.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.657+0000 7fe96ffff640 1 -- 172.21.15.62:0/4104137311 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe98000a640 con 0x7fe9880b9d90 2024-04-18T10:06:58.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.657+0000 7fe98db58640 1 -- 172.21.15.62:0/4104137311 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe988070880 con 0x7fe9880b9d90 2024-04-18T10:06:58.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.657+0000 7fe96ffff640 1 -- 172.21.15.62:0/4104137311 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98000aa30 con 0x7fe9880b9d90 2024-04-18T10:06:58.659 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.657+0000 7fe98db58640 1 -- 172.21.15.62:0/4104137311 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe988142300 con 0x7fe9880b9d90 2024-04-18T10:06:58.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.657+0000 7fe98db58640 1 -- 172.21.15.62:0/4104137311 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe988068b90 con 0x7fe9880b9d90 2024-04-18T10:06:58.660 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.657+0000 7fe96ffff640 1 -- 172.21.15.62:0/4104137311 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe98001b070 con 0x7fe9880b9d90 2024-04-18T10:06:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.657+0000 7fe96ffff640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe9740683b0 0x7fe97406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.657+0000 7fe96ffff640 1 -- 172.21.15.62:0/4104137311 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fe980085c50 con 0x7fe9880b9d90 2024-04-18T10:06:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.657+0000 7fe9877fe640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe9740683b0 0x7fe97406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.661 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.657+0000 7fe9877fe640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe9740683b0 0x7fe97406a870 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe97800b930 tx=0x7fe978008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:58.662 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.661+0000 7fe96ffff640 1 -- 172.21.15.62:0/4104137311 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe98008b020 con 0x7fe9880b9d90 2024-04-18T10:06:58.822 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.821+0000 7fe98db58640 1 -- 172.21.15.62:0/4104137311 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe9880af250 con 0x7fe9740683b0 2024-04-18T10:06:58.824 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe96ffff640 1 -- 172.21.15.62:0/4104137311 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13485 (secure 0 0 0) 0x7fe9880af250 con 0x7fe9740683b0 2024-04-18T10:06:58.825 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:58.825 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-18T10:06:58.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe98db58640 1 -- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe9740683b0 msgr2=0x7fe97406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe98db58640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe9740683b0 0x7fe97406a870 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe97800b930 tx=0x7fe978008040 comp rx=0 tx=0).stop 2024-04-18T10:06:58.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe98db58640 1 -- 172.21.15.62:0/4104137311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9880b9d90 msgr2=0x7fe9880720a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe98db58640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9880b9d90 0x7fe9880720a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe980001570 tx=0x7fe98000cca0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe98db58640 1 -- 172.21.15.62:0/4104137311 shutdown_connections 2024-04-18T10:06:58.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe98db58640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fe9740683b0 0x7fe97406a870 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.827 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe98db58640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9880b9d90 0x7fe9880720a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe98db58640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe98809be30 0x7fe9880727e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe98db58640 1 --2- 172.21.15.62:0/4104137311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe98806dc90 0x7fe988072d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe98db58640 1 -- 172.21.15.62:0/4104137311 >> 172.21.15.62:0/4104137311 conn(0x7fe98806c0d0 msgr2=0x7fe9880afe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:58.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe98db58640 1 -- 172.21.15.62:0/4104137311 shutdown_connections 2024-04-18T10:06:58.828 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.825+0000 7fe98db58640 1 -- 172.21.15.62:0/4104137311 wait complete. 2024-04-18T10:06:58.843 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-04-18T10:06:57.144780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":220188,"kb_used_data":4024,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754532,"statfs":{"total":773094113280,"available":772868640768,"internally_reserved":0,"allocated":4120576,"data_stored":2312437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2851,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-46,"num_write_kb":-15,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002027"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":192,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-18T10:06:56.300560+0000","last_change":"2024-04-18T10:05:49.183850+0000","last_active":"2024-04-18T10:06:56.300560+0000","last_peered":"2024-04-18T10:06:56.300560+0000","last_clean":"2024-04-18T10:06:56.300560+0000","last_became_active":"2024-04-18T10:05:49.181471+0000","last_became_peered":"2024-04-18T10:05:49.181471+0000","last_unstale":"2024-04-18T10:06:56.300560+0000","last_undegraded":"2024-04-18T10:06:56.300560+0000","last_fullsized":"2024-04-18T10:06:56.300560+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_clean_scrub_stamp":"2024-04-18T10:05:47.511486+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:06:51.214547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343880,"statfs":{"total":96636764160,"available":96608133120,"internally_reserved":0,"allocated":958464,"data_stored":731135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27956,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343884,"statfs":{"total":96636764160,"available":96608137216,"internally_reserved":0,"allocated":958464,"data_stored":731135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T10:06:58.843 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-18T10:06:58.941 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-18T10:06:58.944 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-18T10:06:58.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.965+0000 7f968c972640 1 Processor -- start 2024-04-18T10:06:58.967 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.965+0000 7f968c972640 1 -- start start 2024-04-18T10:06:58.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f968c972640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96880701d0 0x7f96881486d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f968c972640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9688148c10 0x7f9688141e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f968c972640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96880705b0 con 0x7f9688142350 2024-04-18T10:06:58.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f968c972640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f9688147ef0 con 0x7f9688148c10 2024-04-18T10:06:58.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f968c972640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f968806fc00 con 0x7f96880701d0 2024-04-18T10:06:58.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f9685d9b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9688148c10 0x7f9688141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f9685d9b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f9688148c10 0x7f9688141e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49086/0 (socket says 172.21.15.62:49086) 2024-04-18T10:06:58.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f9685d9b640 1 -- 172.21.15.62:0/676569752 learned_addr learned my addr 172.21.15.62:0/676569752 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:58.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f968659c640 1 --2- 172.21.15.62:0/676569752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96880701d0 0x7f96881486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f9685d9b640 1 -- 172.21.15.62:0/676569752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96880701d0 msgr2=0x7f96881486d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:06:58.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f9685d9b640 1 --2- 172.21.15.62:0/676569752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96880701d0 0x7f96881486d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f9685d9b640 1 -- 172.21.15.62:0/676569752 >> v1:172.21.15.145:6789/0 conn(0x7f9688142350 legacy=0x7f968814b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f9685d9b640 1 -- 172.21.15.62:0/676569752 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96881481b0 con 0x7f9688148c10 2024-04-18T10:06:58.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f9685d9b640 1 --2- 172.21.15.62:0/676569752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9688148c10 0x7f9688141e10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9678009870 tx=0x7f967801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b03226d2986bbfc server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:58.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f968559a640 1 -- 172.21.15.62:0/676569752 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96780b2070 con 0x7f9688148c10 2024-04-18T10:06:58.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f968559a640 1 -- 172.21.15.62:0/676569752 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96780ad640 con 0x7f9688148c10 2024-04-18T10:06:58.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.969+0000 7f968559a640 1 -- 172.21.15.62:0/676569752 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96780ada10 con 0x7f9688148c10 2024-04-18T10:06:58.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 -- 172.21.15.62:0/676569752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9688148c10 msgr2=0x7f9688141e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 --2- 172.21.15.62:0/676569752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9688148c10 0x7f9688141e10 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f9678009870 tx=0x7f967801c560 comp rx=0 tx=0).stop 2024-04-18T10:06:58.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 -- 172.21.15.62:0/676569752 shutdown_connections 2024-04-18T10:06:58.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 --2- 172.21.15.62:0/676569752 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9688148c10 0x7f9688141e10 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 --2- 172.21.15.62:0/676569752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96880701d0 0x7f96881486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 -- 172.21.15.62:0/676569752 >> 172.21.15.62:0/676569752 conn(0x7f96880a0f00 msgr2=0x7f968809aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:58.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 -- 172.21.15.62:0/676569752 shutdown_connections 2024-04-18T10:06:58.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 -- 172.21.15.62:0/676569752 wait complete. 2024-04-18T10:06:58.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 Processor -- start 2024-04-18T10:06:58.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 -- start start 2024-04-18T10:06:58.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96880701d0 0x7f96880a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9688142350 0x7f9688143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9688148c10 0x7f9688144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f96880b9720 con 0x7f96880701d0 2024-04-18T10:06:58.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f96880b95a0 con 0x7f9688142350 2024-04-18T10:06:58.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f9688073640 con 0x7f9688148c10 2024-04-18T10:06:58.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96880701d0 0x7f96880a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968659c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96880701d0 0x7f96880a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49092/0 (socket says 172.21.15.62:49092) 2024-04-18T10:06:58.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968659c640 1 -- 172.21.15.62:0/4157878567 learned_addr learned my addr 172.21.15.62:0/4157878567 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:06:58.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f9686d9d640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9688148c10 0x7f9688144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f9685d9b640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9688142350 0x7f9688143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968659c640 1 -- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9688148c10 msgr2=0x7f9688144640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968659c640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9688148c10 0x7f9688144640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968659c640 1 -- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9688142350 msgr2=0x7f9688143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:58.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968659c640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9688142350 0x7f9688143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:58.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968659c640 1 -- 172.21.15.62:0/4157878567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9688143b60 con 0x7f96880701d0 2024-04-18T10:06:58.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f9686d9d640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9688148c10 0x7f9688144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:06:58.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968659c640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96880701d0 0x7f96880a0150 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9670009db0 tx=0x7f967000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:58.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f967effd640 1 -- 172.21.15.62:0/4157878567 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9670018020 con 0x7f96880701d0 2024-04-18T10:06:58.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f967effd640 1 -- 172.21.15.62:0/4157878567 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9670014480 con 0x7f96880701d0 2024-04-18T10:06:58.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 -- 172.21.15.62:0/4157878567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9688143e40 con 0x7f96880701d0 2024-04-18T10:06:58.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f967effd640 1 -- 172.21.15.62:0/4157878567 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9670014850 con 0x7f96880701d0 2024-04-18T10:06:58.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 -- 172.21.15.62:0/4157878567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9688143670 con 0x7f96880701d0 2024-04-18T10:06:58.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.973+0000 7f968c972640 1 -- 172.21.15.62:0/4157878567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9688069230 con 0x7f96880701d0 2024-04-18T10:06:58.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.977+0000 7f967effd640 1 -- 172.21.15.62:0/4157878567 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f967001b020 con 0x7f96880701d0 2024-04-18T10:06:58.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.977+0000 7f967effd640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f966c068380 0x7f966c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:06:58.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.977+0000 7f967effd640 1 -- 172.21.15.62:0/4157878567 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f96700871d0 con 0x7f96880701d0 2024-04-18T10:06:58.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.977+0000 7f9685d9b640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f966c068380 0x7f966c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:06:58.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.977+0000 7f9685d9b640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f966c068380 0x7f966c06a840 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f96780af000 tx=0x7f9678008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:06:58.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:58.981+0000 7f967effd640 1 -- 172.21.15.62:0/4157878567 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9670022030 con 0x7f96880701d0 2024-04-18T10:06:59.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.137+0000 7f968c972640 1 -- 172.21.15.62:0/4157878567 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9688140100 con 0x7f96880701d0 2024-04-18T10:06:59.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f967effd640 1 -- 172.21.15.62:0/4157878567 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+13008 (secure 0 0 0) 0x7f9670085e20 con 0x7f96880701d0 2024-04-18T10:06:59.140 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:06:59.141 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":33,"fsid":"26864971-6592-407b-b492-2c0e3b7ea567","created":"2024-04-18T10:05:34.342543+0000","modified":"2024-04-18T10:06:56.256834+0000","last_up_change":"2024-04-18T10:05:45.468204+0000","last_in_change":"2024-04-18T10:05:36.635206+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-18T10:05:47.165426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7cdcd566-e84d-4ada-8ca8-f3a4f0bb6d50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6817","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6819","nonce":3805589275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6821","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6824","nonce":3805589275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6831","nonce":3805589275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3805589275},{"type":"v1","addr":"172.21.15.62:6828","nonce":3805589275}]},"public_addr":"172.21.15.62:6819/3805589275","cluster_addr":"172.21.15.62:6824/3805589275","heartbeat_back_addr":"172.21.15.62:6831/3805589275","heartbeat_front_addr":"172.21.15.62:6828/3805589275","state":["exists","up"]},{"osd":1,"uuid":"71e116c5-995e-4a75-8218-2f5003e6a6a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6801","nonce":4072301756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6803","nonce":4072301756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6807","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6808","nonce":4072301756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":4072301756},{"type":"v1","addr":"172.21.15.62:6805","nonce":4072301756}]},"public_addr":"172.21.15.62:6801/4072301756","cluster_addr":"172.21.15.62:6803/4072301756","heartbeat_back_addr":"172.21.15.62:6808/4072301756","heartbeat_front_addr":"172.21.15.62:6805/4072301756","state":["exists","up"]},{"osd":2,"uuid":"e6257209-4245-4fde-9aa9-461258b78577","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6818","nonce":1732263675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6822","nonce":1732263675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6827","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6829","nonce":1732263675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6823","nonce":1732263675},{"type":"v1","addr":"172.21.15.62:6825","nonce":1732263675}]},"public_addr":"172.21.15.62:6818/1732263675","cluster_addr":"172.21.15.62:6822/1732263675","heartbeat_back_addr":"172.21.15.62:6829/1732263675","heartbeat_front_addr":"172.21.15.62:6825/1732263675","state":["exists","up"]},{"osd":3,"uuid":"80fad71f-3ef3-46ab-a81c-104077404f09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6809","nonce":1675715975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6811","nonce":1675715975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6815","nonce":1675715975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":1675715975},{"type":"v1","addr":"172.21.15.62:6813","nonce":1675715975}]},"public_addr":"172.21.15.62:6809/1675715975","cluster_addr":"172.21.15.62:6811/1675715975","heartbeat_back_addr":"172.21.15.62:6815/1675715975","heartbeat_front_addr":"172.21.15.62:6813/1675715975","state":["exists","up"]},{"osd":4,"uuid":"5a341da7-aaa2-4439-956a-6e9dc824aa38","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6817","nonce":4248694380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6819","nonce":4248694380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6823","nonce":4248694380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4248694380},{"type":"v1","addr":"172.21.15.145:6821","nonce":4248694380}]},"public_addr":"172.21.15.145:6817/4248694380","cluster_addr":"172.21.15.145:6819/4248694380","heartbeat_back_addr":"172.21.15.145:6823/4248694380","heartbeat_front_addr":"172.21.15.145:6821/4248694380","state":["exists","up"]},{"osd":5,"uuid":"5d0d4db5-bf9b-42d3-9987-c23ce0c96ca9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6825","nonce":3413756159}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6827","nonce":3413756159}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6831","nonce":3413756159}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3413756159},{"type":"v1","addr":"172.21.15.145:6829","nonce":3413756159}]},"public_addr":"172.21.15.145:6825/3413756159","cluster_addr":"172.21.15.145:6827/3413756159","heartbeat_back_addr":"172.21.15.145:6831/3413756159","heartbeat_front_addr":"172.21.15.145:6829/3413756159","state":["exists","up"]},{"osd":6,"uuid":"20bac992-d716-415f-accc-1acbb8874d44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6801","nonce":2070684678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6803","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6805","nonce":2070684678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6811","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6813","nonce":2070684678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6807","nonce":2070684678},{"type":"v1","addr":"172.21.15.145:6809","nonce":2070684678}]},"public_addr":"172.21.15.145:6801/2070684678","cluster_addr":"172.21.15.145:6805/2070684678","heartbeat_back_addr":"172.21.15.145:6813/2070684678","heartbeat_front_addr":"172.21.15.145:6809/2070684678","state":["exists","up"]},{"osd":7,"uuid":"7de29c9e-17ce-496a-bada-ec9b482f8f04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6804","nonce":69473053}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6808","nonce":69473053}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6815","nonce":69473053}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":69473053},{"type":"v1","addr":"172.21.15.145:6812","nonce":69473053}]},"public_addr":"172.21.15.145:6804/69473053","cluster_addr":"172.21.15.145:6808/69473053","heartbeat_back_addr":"172.21.15.145:6815/69473053","heartbeat_front_addr":"172.21.15.145:6812/69473053","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.202172+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.140124+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.254470+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.027364+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.270709+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.002269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.194695+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-18T10:05:42.128974+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6833/2971073685":"2024-04-19T10:06:53.179269+0000","172.21.15.62:6832/2971073685":"2024-04-19T10:06:53.179269+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-18T10:06:59.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f968c972640 1 -- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f966c068380 msgr2=0x7f966c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:59.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f968c972640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f966c068380 0x7f966c06a840 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f96780af000 tx=0x7f9678008040 comp rx=0 tx=0).stop 2024-04-18T10:06:59.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f968c972640 1 -- 172.21.15.62:0/4157878567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96880701d0 msgr2=0x7f96880a0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:06:59.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f968c972640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96880701d0 0x7f96880a0150 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9670009db0 tx=0x7f967000c490 comp rx=0 tx=0).stop 2024-04-18T10:06:59.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f968c972640 1 -- 172.21.15.62:0/4157878567 shutdown_connections 2024-04-18T10:06:59.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f968c972640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f966c068380 0x7f966c06a840 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:59.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f968c972640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9688148c10 0x7f9688144640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:59.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f968c972640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9688142350 0x7f9688143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:59.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f968c972640 1 --2- 172.21.15.62:0/4157878567 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f96880701d0 0x7f96880a0150 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:06:59.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f968c972640 1 -- 172.21.15.62:0/4157878567 >> 172.21.15.62:0/4157878567 conn(0x7f96880a0f00 msgr2=0x7f96880ba610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:06:59.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f968c972640 1 -- 172.21.15.62:0/4157878567 shutdown_connections 2024-04-18T10:06:59.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:06:59.141+0000 7f968c972640 1 -- 172.21.15.62:0/4157878567 wait complete. 2024-04-18T10:07:00.159 INFO:tasks.ceph:Scrubbing osd.0 2024-04-18T10:07:00.160 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-18T10:07:00.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.297+0000 7f6747b62640 1 Processor -- start 2024-04-18T10:07:00.299 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.297+0000 7f6747b62640 1 -- start start 2024-04-18T10:07:00.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f67400bbf20 0x7f67400b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.300 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f67400b7c50 0x7f67400b63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67400bc300 con 0x7f6740141d10 2024-04-18T10:07:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f674006e8c0 con 0x7f67400b7c50 2024-04-18T10:07:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f67400ae060 con 0x7f67400bbf20 2024-04-18T10:07:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f67450fd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f67400b7c50 0x7f67400b63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f67450fd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f67400b7c50 0x7f67400b63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49094/0 (socket says 172.21.15.62:49094) 2024-04-18T10:07:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f67450fd640 1 -- 172.21.15.62:0/2216217216 learned_addr learned my addr 172.21.15.62:0/2216217216 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f67458fe640 1 --2- 172.21.15.62:0/2216217216 >> v2:172.21.15.145:3300/0 conn(0x7f67400bbf20 0x7f67400b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.301 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f67450fd640 1 -- 172.21.15.62:0/2216217216 >> v2:172.21.15.145:3300/0 conn(0x7f67400bbf20 msgr2=0x7f67400b5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:00.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f67450fd640 1 --2- 172.21.15.62:0/2216217216 >> v2:172.21.15.145:3300/0 conn(0x7f67400bbf20 0x7f67400b5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f67450fd640 1 -- 172.21.15.62:0/2216217216 >> v1:172.21.15.145:6789/0 conn(0x7f6740141d10 legacy=0x7f6740142130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f67450fd640 1 -- 172.21.15.62:0/2216217216 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67400adca0 con 0x7f67400b7c50 2024-04-18T10:07:00.302 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f67450fd640 1 --2- 172.21.15.62:0/2216217216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400b7c50 0x7f67400b63f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f674006ea40 tx=0x7f673001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e28066b32f3dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f67448fc640 1 -- 172.21.15.62:0/2216217216 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67300b0070 con 0x7f67400b7c50 2024-04-18T10:07:00.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f67448fc640 1 -- 172.21.15.62:0/2216217216 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67300ab600 con 0x7f67400b7c50 2024-04-18T10:07:00.303 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f67448fc640 1 -- 172.21.15.62:0/2216217216 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67300ab9d0 con 0x7f67400b7c50 2024-04-18T10:07:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 -- 172.21.15.62:0/2216217216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400b7c50 msgr2=0x7f67400b63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 --2- 172.21.15.62:0/2216217216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400b7c50 0x7f67400b63f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f674006ea40 tx=0x7f673001c520 comp rx=0 tx=0).stop 2024-04-18T10:07:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 -- 172.21.15.62:0/2216217216 shutdown_connections 2024-04-18T10:07:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 --2- 172.21.15.62:0/2216217216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67400b7c50 0x7f67400b63f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 --2- 172.21.15.62:0/2216217216 >> v2:172.21.15.145:3300/0 conn(0x7f67400bbf20 0x7f67400b5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 -- 172.21.15.62:0/2216217216 >> 172.21.15.62:0/2216217216 conn(0x7f6740067de0 msgr2=0x7f67400bb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 -- 172.21.15.62:0/2216217216 shutdown_connections 2024-04-18T10:07:00.304 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.301+0000 7f6747b62640 1 -- 172.21.15.62:0/2216217216 wait complete. 2024-04-18T10:07:00.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f6747b62640 1 Processor -- start 2024-04-18T10:07:00.306 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f6747b62640 1 -- start start 2024-04-18T10:07:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f6747b62640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67400bbf20 0x7f6740147050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f6747b62640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6740141d10 0x7f6740145d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f6747b62640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6740146290 0x7f674013cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f6747b62640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6740155330 con 0x7f6740141d10 2024-04-18T10:07:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f6747b62640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f67401551b0 con 0x7f67400bbf20 2024-04-18T10:07:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f6747b62640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f67401554b0 con 0x7f6740146290 2024-04-18T10:07:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67450fd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6740141d10 0x7f6740145d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.307 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67450fd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6740141d10 0x7f6740145d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49104/0 (socket says 172.21.15.62:49104) 2024-04-18T10:07:00.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67450fd640 1 -- 172.21.15.62:0/2134182843 learned_addr learned my addr 172.21.15.62:0/2134182843 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:00.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67458fe640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67400bbf20 0x7f6740147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67450fd640 1 -- 172.21.15.62:0/2134182843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6740146290 msgr2=0x7f674013cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67450fd640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6740146290 0x7f674013cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67450fd640 1 -- 172.21.15.62:0/2134182843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67400bbf20 msgr2=0x7f6740147050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67450fd640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67400bbf20 0x7f6740147050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67450fd640 1 -- 172.21.15.62:0/2134182843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6740146710 con 0x7f6740141d10 2024-04-18T10:07:00.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67458fe640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67400bbf20 0x7f6740147050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:00.308 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67450fd640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6740141d10 0x7f6740145d50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f673001abd0 tx=0x7f67300192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67367fc640 1 -- 172.21.15.62:0/2134182843 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67300bb070 con 0x7f6740141d10 2024-04-18T10:07:00.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67367fc640 1 -- 172.21.15.62:0/2134182843 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67300b6430 con 0x7f6740141d10 2024-04-18T10:07:00.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f6747b62640 1 -- 172.21.15.62:0/2134182843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674013f230 con 0x7f6740141d10 2024-04-18T10:07:00.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67367fc640 1 -- 172.21.15.62:0/2134182843 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67300b0040 con 0x7f6740141d10 2024-04-18T10:07:00.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f6747b62640 1 -- 172.21.15.62:0/2134182843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674013e0d0 con 0x7f6740141d10 2024-04-18T10:07:00.309 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f6747b62640 1 -- 172.21.15.62:0/2134182843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6740068ca0 con 0x7f6740141d10 2024-04-18T10:07:00.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67367fc640 1 -- 172.21.15.62:0/2134182843 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f67300be070 con 0x7f6740141d10 2024-04-18T10:07:00.310 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.305+0000 7f67367fc640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6724068380 0x7f672406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.309+0000 7f67367fc640 1 -- 172.21.15.62:0/2134182843 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f673011fdb0 con 0x7f6740141d10 2024-04-18T10:07:00.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.309+0000 7f67367fc640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] conn(0x7f672406e240 0x7f6724070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.309+0000 7f67367fc640 1 -- 172.21.15.62:0/2134182843 --> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f67300b2e50 con 0x7f672406e240 2024-04-18T10:07:00.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.309+0000 7f67367fc640 1 -- 172.21.15.62:0/2134182843 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f67301224b0 con 0x7f6740141d10 2024-04-18T10:07:00.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.309+0000 7f67458fe640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6724068380 0x7f672406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.311 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.309+0000 7f67460ff640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] conn(0x7f672406e240 0x7f6724070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.309+0000 7f67460ff640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] conn(0x7f672406e240 0x7f6724070660 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.309+0000 7f67458fe640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6724068380 0x7f672406a840 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f673c009ae0 tx=0x7f673c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.312 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.309+0000 7f67367fc640 1 -- 172.21.15.62:0/2134182843 <== osd.0 v2:172.21.15.62:6817/3805589275 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f67300b2e50 con 0x7f672406e240 2024-04-18T10:07:00.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f6747b62640 1 -- 172.21.15.62:0/2134182843 --> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6740148550 con 0x7f672406e240 2024-04-18T10:07:00.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f67367fc640 1 -- 172.21.15.62:0/2134182843 <== osd.0 v2:172.21.15.62:6817/3805589275 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6740148550 con 0x7f672406e240 2024-04-18T10:07:00.323 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:07:00.323 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T10:07:00.323 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:07:00.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f6747b62640 1 -- 172.21.15.62:0/2134182843 >> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] conn(0x7f672406e240 msgr2=0x7f6724070660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f6747b62640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.62:6817/3805589275,v1:172.21.15.62:6819/3805589275] conn(0x7f672406e240 0x7f6724070660 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f6747b62640 1 -- 172.21.15.62:0/2134182843 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6724068380 msgr2=0x7f672406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f6747b62640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f6724068380 0x7f672406a840 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f673c009ae0 tx=0x7f673c008040 comp rx=0 tx=0).stop 2024-04-18T10:07:00.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f6747b62640 1 -- 172.21.15.62:0/2134182843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6740141d10 msgr2=0x7f6740145d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f6747b62640 1 --2- 172.21.15.62:0/2134182843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6740141d10 0x7f6740145d50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f673001abd0 tx=0x7f67300192c0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.324 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f67460ff640 1 -- 172.21.15.62:0/2134182843 reap_dead start 2024-04-18T10:07:00.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f6747b62640 1 -- 172.21.15.62:0/2134182843 shutdown_connections 2024-04-18T10:07:00.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f6747b62640 1 -- 172.21.15.62:0/2134182843 >> 172.21.15.62:0/2134182843 conn(0x7f6740067de0 msgr2=0x7f6740074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:00.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f6747b62640 1 -- 172.21.15.62:0/2134182843 shutdown_connections 2024-04-18T10:07:00.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.321+0000 7f6747b62640 1 -- 172.21.15.62:0/2134182843 wait complete. 2024-04-18T10:07:00.339 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-18T10:07:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5f165640 1 Processor -- start 2024-04-18T10:07:00.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5f165640 1 -- start start 2024-04-18T10:07:00.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5f165640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf580aeda0 0x7fcf581498b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5f165640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf58149df0 0x7fcf581504d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5f165640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf58067e50 con 0x7fcf58067a70 2024-04-18T10:07:00.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5f165640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fcf58146b60 con 0x7fcf58149df0 2024-04-18T10:07:00.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5f165640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fcf58149140 con 0x7fcf580aeda0 2024-04-18T10:07:00.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5d702640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf58149df0 0x7fcf581504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.456 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5d702640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcf58149df0 0x7fcf581504d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49108/0 (socket says 172.21.15.62:49108) 2024-04-18T10:07:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5d702640 1 -- 172.21.15.62:0/4268166079 learned_addr learned my addr 172.21.15.62:0/4268166079 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf57fff640 1 --2- 172.21.15.62:0/4268166079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf580aeda0 0x7fcf581498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5d702640 1 -- 172.21.15.62:0/4268166079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf580aeda0 msgr2=0x7fcf581498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5d702640 1 --2- 172.21.15.62:0/4268166079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf580aeda0 0x7fcf581498b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5d702640 1 -- 172.21.15.62:0/4268166079 >> v1:172.21.15.145:6789/0 conn(0x7fcf58067a70 legacy=0x7fcf5813faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf5d702640 1 -- 172.21.15.62:0/4268166079 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf580b0040 con 0x7fcf58149df0 2024-04-18T10:07:00.457 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.453+0000 7fcf57fff640 1 --2- 172.21.15.62:0/4268166079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf580aeda0 0x7fcf581498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:07:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.457+0000 7fcf5d702640 1 --2- 172.21.15.62:0/4268166079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf58149df0 0x7fcf581504d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009e50 tx=0x7fcf5001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b052ca295594e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.457+0000 7fcf577fe640 1 -- 172.21.15.62:0/4268166079 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf500a5860 con 0x7fcf58149df0 2024-04-18T10:07:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.457+0000 7fcf577fe640 1 -- 172.21.15.62:0/4268166079 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf500a5a00 con 0x7fcf58149df0 2024-04-18T10:07:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.457+0000 7fcf577fe640 1 -- 172.21.15.62:0/4268166079 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf500a5dd0 con 0x7fcf58149df0 2024-04-18T10:07:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.457+0000 7fcf5f165640 1 -- 172.21.15.62:0/4268166079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf58149df0 msgr2=0x7fcf581504d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.459 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.457+0000 7fcf5f165640 1 --2- 172.21.15.62:0/4268166079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf58149df0 0x7fcf581504d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcf50009e50 tx=0x7fcf5001c9a0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.457+0000 7fcf5f165640 1 -- 172.21.15.62:0/4268166079 shutdown_connections 2024-04-18T10:07:00.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.457+0000 7fcf5f165640 1 --2- 172.21.15.62:0/4268166079 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf58149df0 0x7fcf581504d0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.457+0000 7fcf5f165640 1 --2- 172.21.15.62:0/4268166079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf580aeda0 0x7fcf581498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.457+0000 7fcf5f165640 1 -- 172.21.15.62:0/4268166079 >> 172.21.15.62:0/4268166079 conn(0x7fcf580b9c80 msgr2=0x7fcf580ba090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:00.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.457+0000 7fcf5f165640 1 -- 172.21.15.62:0/4268166079 shutdown_connections 2024-04-18T10:07:00.460 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.457+0000 7fcf5f165640 1 -- 172.21.15.62:0/4268166079 wait complete. 2024-04-18T10:07:00.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5f165640 1 Processor -- start 2024-04-18T10:07:00.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5f165640 1 -- start start 2024-04-18T10:07:00.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5f165640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf58067a70 0x7fcf580b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5f165640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf580aeda0 0x7fcf581decb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.463 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5f165640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf58149df0 0x7fcf58139d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5f165640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcf58153230 con 0x7fcf58067a70 2024-04-18T10:07:00.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5f165640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fcf581401b0 con 0x7fcf580aeda0 2024-04-18T10:07:00.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5f165640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fcf581533b0 con 0x7fcf58149df0 2024-04-18T10:07:00.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5cf01640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf58067a70 0x7fcf580b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5cf01640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf58067a70 0x7fcf580b1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49114/0 (socket says 172.21.15.62:49114) 2024-04-18T10:07:00.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5cf01640 1 -- 172.21.15.62:0/2657388133 learned_addr learned my addr 172.21.15.62:0/2657388133 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:00.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf57fff640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf580aeda0 0x7fcf581decb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.464 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5d702640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf58149df0 0x7fcf58139d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5cf01640 1 -- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf58149df0 msgr2=0x7fcf58139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5cf01640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf58149df0 0x7fcf58139d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5cf01640 1 -- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf580aeda0 msgr2=0x7fcf581decb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5cf01640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf580aeda0 0x7fcf581decb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5cf01640 1 -- 172.21.15.62:0/2657388133 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf581de510 con 0x7fcf58067a70 2024-04-18T10:07:00.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5d702640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf58149df0 0x7fcf58139d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:00.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf57fff640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf580aeda0 0x7fcf581decb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:00.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5cf01640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf58067a70 0x7fcf580b1aa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fcf4800c370 tx=0x7fcf4800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.465 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf557fa640 1 -- 172.21.15.62:0/2657388133 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf480052d0 con 0x7fcf58067a70 2024-04-18T10:07:00.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf557fa640 1 -- 172.21.15.62:0/2657388133 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf48005470 con 0x7fcf58067a70 2024-04-18T10:07:00.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf557fa640 1 -- 172.21.15.62:0/2657388133 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf48007260 con 0x7fcf58067a70 2024-04-18T10:07:00.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5f165640 1 -- 172.21.15.62:0/2657388133 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf580b3c30 con 0x7fcf58067a70 2024-04-18T10:07:00.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5f165640 1 -- 172.21.15.62:0/2657388133 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf581de790 con 0x7fcf58067a70 2024-04-18T10:07:00.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf5f165640 1 -- 172.21.15.62:0/2657388133 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf58068b20 con 0x7fcf58067a70 2024-04-18T10:07:00.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf557fa640 1 -- 172.21.15.62:0/2657388133 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcf48007400 con 0x7fcf58067a70 2024-04-18T10:07:00.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf557fa640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf44068380 0x7fcf4406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf557fa640 1 -- 172.21.15.62:0/2657388133 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fcf48085a40 con 0x7fcf58067a70 2024-04-18T10:07:00.466 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.461+0000 7fcf57fff640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf44068380 0x7fcf4406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.465+0000 7fcf57fff640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf44068380 0x7fcf4406a840 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcf58140080 tx=0x7fcf4c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.467 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.465+0000 7fcf557fa640 1 -- 172.21.15.62:0/2657388133 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcf581de510 con 0x7fcf58067a70 2024-04-18T10:07:00.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.629+0000 7fcf5f165640 1 -- 172.21.15.62:0/2657388133 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fcf580734d0 con 0x7fcf44068380 2024-04-18T10:07:00.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf557fa640 1 -- 172.21.15.62:0/2657388133 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fcf580734d0 con 0x7fcf44068380 2024-04-18T10:07:00.632 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 0 to deep-scrub 2024-04-18T10:07:00.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf5f165640 1 -- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf44068380 msgr2=0x7fcf4406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf5f165640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf44068380 0x7fcf4406a840 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fcf58140080 tx=0x7fcf4c008040 comp rx=0 tx=0).stop 2024-04-18T10:07:00.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf5f165640 1 -- 172.21.15.62:0/2657388133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf58067a70 msgr2=0x7fcf580b1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.634 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf5f165640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf58067a70 0x7fcf580b1aa0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fcf4800c370 tx=0x7fcf4800c830 comp rx=0 tx=0).stop 2024-04-18T10:07:00.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf5f165640 1 -- 172.21.15.62:0/2657388133 shutdown_connections 2024-04-18T10:07:00.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf5f165640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf44068380 0x7fcf4406a840 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf5f165640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf58149df0 0x7fcf58139d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf5f165640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf580aeda0 0x7fcf581decb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf5f165640 1 --2- 172.21.15.62:0/2657388133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf58067a70 0x7fcf580b1aa0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf5f165640 1 -- 172.21.15.62:0/2657388133 >> 172.21.15.62:0/2657388133 conn(0x7fcf580b9c80 msgr2=0x7fcf580b7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:00.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf5f165640 1 -- 172.21.15.62:0/2657388133 shutdown_connections 2024-04-18T10:07:00.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.633+0000 7fcf5f165640 1 -- 172.21.15.62:0/2657388133 wait complete. 2024-04-18T10:07:00.651 INFO:tasks.ceph:Scrubbing osd.1 2024-04-18T10:07:00.651 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-18T10:07:00.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f8278c98640 1 Processor -- start 2024-04-18T10:07:00.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f8278c98640 1 -- start start 2024-04-18T10:07:00.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f8278c98640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f82740b3e20 0x7f82741461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f8278c98640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82740ba170 con 0x7f827406d920 2024-04-18T10:07:00.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f8278c98640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f827409cca0 con 0x7f82740b9d90 2024-04-18T10:07:00.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f8278c98640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f8274149930 con 0x7f82740b3e20 2024-04-18T10:07:00.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f8272d9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f827406d920 0x7f8274150470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48764/0 (socket says 172.21.15.62:48764) 2024-04-18T10:07:00.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f8272d9d640 1 -- 172.21.15.62:0/2568126722 learned_addr learned my addr 172.21.15.62:0/2568126722 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:07:00.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f8271d9b640 1 --2- 172.21.15.62:0/2568126722 >> v2:172.21.15.145:3300/0 conn(0x7f82740b3e20 0x7f82741461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418749632 0 0) 0x7f82740ba170 con 0x7f827406d920 2024-04-18T10:07:00.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f825c003530 con 0x7f827406d920 2024-04-18T10:07:00.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3448312570 0 0) 0x7f825c003530 con 0x7f827406d920 2024-04-18T10:07:00.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 >> v2:172.21.15.145:3300/0 conn(0x7f82740b3e20 msgr2=0x7f82741461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:00.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 --2- 172.21.15.62:0/2568126722 >> v2:172.21.15.145:3300/0 conn(0x7f82740b3e20 0x7f82741461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 >> v1:172.21.15.145:6789/0 conn(0x7f82740b9d90 legacy=0x7f827406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8274149c10 con 0x7f827406d920 2024-04-18T10:07:00.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f826c001a00 con 0x7f827406d920 2024-04-18T10:07:00.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 >> v1:172.21.15.62:6789/0 conn(0x7f827406d920 legacy=0x7f8274150470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 --2- 172.21.15.62:0/2568126722 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c0039c0 0x7f825c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 --2- 172.21.15.62:0/2568126722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f825c004810 0x7f825c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 --2- 172.21.15.62:0/2568126722 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c0051d0 0x7f825c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f8274149930 con 0x7f825c004810 2024-04-18T10:07:00.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f825c00c170 con 0x7f825c0051d0 2024-04-18T10:07:00.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.773+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f825c00c370 con 0x7f825c0039c0 2024-04-18T10:07:00.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8271d9b640 1 --2- 172.21.15.62:0/2568126722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f825c004810 0x7f825c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8272d9d640 1 --2- 172.21.15.62:0/2568126722 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c0051d0 0x7f825c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f827259c640 1 --2- 172.21.15.62:0/2568126722 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c0039c0 0x7f825c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8271d9b640 1 -- 172.21.15.62:0/2568126722 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c0039c0 msgr2=0x7f825c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:00.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8271d9b640 1 --2- 172.21.15.62:0/2568126722 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f825c0039c0 0x7f825c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8271d9b640 1 -- 172.21.15.62:0/2568126722 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c0051d0 msgr2=0x7f825c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:00.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8271d9b640 1 --2- 172.21.15.62:0/2568126722 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c0051d0 0x7f825c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8271d9b640 1 -- 172.21.15.62:0/2568126722 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c003530 con 0x7f825c004810 2024-04-18T10:07:00.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8272d9d640 1 -- 172.21.15.62:0/2568126722 reap_dead start 2024-04-18T10:07:00.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8271d9b640 1 --2- 172.21.15.62:0/2568126722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f825c004810 0x7f825c004c90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8268009ae0 tx=0x7f826801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db427f34e85455b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82680096b0 con 0x7f825c004810 2024-04-18T10:07:00.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82680a6450 con 0x7f825c004810 2024-04-18T10:07:00.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f827159a640 1 -- 172.21.15.62:0/2568126722 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82680a6840 con 0x7f825c004810 2024-04-18T10:07:00.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8278c98640 1 -- 172.21.15.62:0/2568126722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f825c004810 msgr2=0x7f825c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8278c98640 1 --2- 172.21.15.62:0/2568126722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f825c004810 0x7f825c004c90 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8268009ae0 tx=0x7f826801ca70 comp rx=0 tx=0).stop 2024-04-18T10:07:00.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8278c98640 1 -- 172.21.15.62:0/2568126722 shutdown_connections 2024-04-18T10:07:00.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8278c98640 1 --2- 172.21.15.62:0/2568126722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f825c004810 0x7f825c004c90 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8278c98640 1 -- 172.21.15.62:0/2568126722 >> 172.21.15.62:0/2568126722 conn(0x7f827406c0d0 msgr2=0x7f82740b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:00.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8278c98640 1 -- 172.21.15.62:0/2568126722 shutdown_connections 2024-04-18T10:07:00.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.777+0000 7f8278c98640 1 -- 172.21.15.62:0/2568126722 wait complete. 2024-04-18T10:07:00.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8278c98640 1 Processor -- start 2024-04-18T10:07:00.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8278c98640 1 -- start start 2024-04-18T10:07:00.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8278c98640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c004810 0x7f8274071090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8278c98640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8274071fd0 0x7f82741df220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8278c98640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82741df760 0x7f8274141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8278c98640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f82741459e0 con 0x7f82741df760 2024-04-18T10:07:00.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8278c98640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f827409c020 con 0x7f825c004810 2024-04-18T10:07:00.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8278c98640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f8274145b60 con 0x7f8274071fd0 2024-04-18T10:07:00.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8272d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82741df760 0x7f8274141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8272d9d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82741df760 0x7f8274141ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49124/0 (socket says 172.21.15.62:49124) 2024-04-18T10:07:00.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8272d9d640 1 -- 172.21.15.62:0/2511786964 learned_addr learned my addr 172.21.15.62:0/2511786964 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:00.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f827259c640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c004810 0x7f8274071090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8271d9b640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8274071fd0 0x7f82741df220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8272d9d640 1 -- 172.21.15.62:0/2511786964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8274071fd0 msgr2=0x7f82741df220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8272d9d640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8274071fd0 0x7f82741df220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8272d9d640 1 -- 172.21.15.62:0/2511786964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c004810 msgr2=0x7f8274071090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8272d9d640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f825c004810 0x7f8274071090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8272d9d640 1 -- 172.21.15.62:0/2511786964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82741d1570 con 0x7f82741df760 2024-04-18T10:07:00.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8272d9d640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82741df760 0x7f8274141ee0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f82740b5ef0 tx=0x7f826c007b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8262ffd640 1 -- 172.21.15.62:0/2511786964 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c00eaa0 con 0x7f82741df760 2024-04-18T10:07:00.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8262ffd640 1 -- 172.21.15.62:0/2511786964 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f826c018070 con 0x7f82741df760 2024-04-18T10:07:00.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8278c98640 1 -- 172.21.15.62:0/2511786964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82740727d0 con 0x7f82741df760 2024-04-18T10:07:00.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8262ffd640 1 -- 172.21.15.62:0/2511786964 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f826c013420 con 0x7f82741df760 2024-04-18T10:07:00.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8278c98640 1 -- 172.21.15.62:0/2511786964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82741d1290 con 0x7f82741df760 2024-04-18T10:07:00.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8278c98640 1 -- 172.21.15.62:0/2511786964 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f82741dfc10 con 0x7f82741df760 2024-04-18T10:07:00.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8262ffd640 1 -- 172.21.15.62:0/2511786964 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f826c0135c0 con 0x7f82741df760 2024-04-18T10:07:00.788 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8262ffd640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8254068380 0x7f825406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8262ffd640 1 -- 172.21.15.62:0/2511786964 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f826c087360 con 0x7f82741df760 2024-04-18T10:07:00.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8262ffd640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] conn(0x7f825406e240 0x7f8254070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8262ffd640 1 -- 172.21.15.62:0/2511786964 --> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f826c006ea0 con 0x7f825406e240 2024-04-18T10:07:00.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8262ffd640 1 -- 172.21.15.62:0/2511786964 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f826c0877f0 con 0x7f82741df760 2024-04-18T10:07:00.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f827259c640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8254068380 0x7f825406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.781+0000 7f8271d9b640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] conn(0x7f825406e240 0x7f8254070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.785+0000 7f8271d9b640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] conn(0x7f825406e240 0x7f8254070660 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.785+0000 7f827259c640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8254068380 0x7f825406a840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8264000c10 tx=0x7f8264009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.785+0000 7f8262ffd640 1 -- 172.21.15.62:0/2511786964 <== osd.1 v2:172.21.15.62:6800/4072301756 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f826c006ea0 con 0x7f825406e240 2024-04-18T10:07:00.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8278c98640 1 -- 172.21.15.62:0/2511786964 --> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8274073540 con 0x7f825406e240 2024-04-18T10:07:00.797 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8262ffd640 1 -- 172.21.15.62:0/2511786964 <== osd.1 v2:172.21.15.62:6800/4072301756 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f82680a6450 con 0x7f825406e240 2024-04-18T10:07:00.797 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:07:00.798 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T10:07:00.798 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:07:00.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8278c98640 1 -- 172.21.15.62:0/2511786964 >> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] conn(0x7f825406e240 msgr2=0x7f8254070660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8278c98640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.62:6800/4072301756,v1:172.21.15.62:6801/4072301756] conn(0x7f825406e240 0x7f8254070660 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8278c98640 1 -- 172.21.15.62:0/2511786964 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8254068380 msgr2=0x7f825406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8278c98640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f8254068380 0x7f825406a840 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8264000c10 tx=0x7f8264009040 comp rx=0 tx=0).stop 2024-04-18T10:07:00.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8278c98640 1 -- 172.21.15.62:0/2511786964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82741df760 msgr2=0x7f8274141ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8278c98640 1 --2- 172.21.15.62:0/2511786964 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82741df760 0x7f8274141ee0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f82740b5ef0 tx=0x7f826c007b70 comp rx=0 tx=0).stop 2024-04-18T10:07:00.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8272d9d640 1 -- 172.21.15.62:0/2511786964 reap_dead start 2024-04-18T10:07:00.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8278c98640 1 -- 172.21.15.62:0/2511786964 shutdown_connections 2024-04-18T10:07:00.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8278c98640 1 -- 172.21.15.62:0/2511786964 >> 172.21.15.62:0/2511786964 conn(0x7f827406c0d0 msgr2=0x7f82740aed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:00.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8278c98640 1 -- 172.21.15.62:0/2511786964 shutdown_connections 2024-04-18T10:07:00.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.797+0000 7f8278c98640 1 -- 172.21.15.62:0/2511786964 wait complete. 2024-04-18T10:07:00.812 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-18T10:07:00.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad09c04640 1 Processor -- start 2024-04-18T10:07:00.922 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad09c04640 1 -- start start 2024-04-18T10:07:00.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad09c04640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0409c4d0 0x7fad0409c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad09c04640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fad040b3eb0 0x7fad040b4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad09c04640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad040b4850 con 0x7fad0406d210 2024-04-18T10:07:00.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad09c04640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fad04098da0 con 0x7fad040b3eb0 2024-04-18T10:07:00.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad09c04640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fad040b5970 con 0x7fad0409c4d0 2024-04-18T10:07:00.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad02ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fad040b3eb0 0x7fad040b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad03fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fad0406d210 0x7fad0406d640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:48770/0 (socket says 172.21.15.62:48770) 2024-04-18T10:07:00.924 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad03fff640 1 -- 172.21.15.62:0/3550194814 learned_addr learned my addr 172.21.15.62:0/3550194814 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:07:00.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad037fe640 1 --2- 172.21.15.62:0/3550194814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0409c4d0 0x7fad0409c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad02ffd640 1 -- 172.21.15.62:0/3550194814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0409c4d0 msgr2=0x7fad0409c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:00.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad02ffd640 1 --2- 172.21.15.62:0/3550194814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0409c4d0 0x7fad0409c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad02ffd640 1 -- 172.21.15.62:0/3550194814 >> v1:172.21.15.62:6789/0 conn(0x7fad0406d210 legacy=0x7fad0406d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad027fc640 1 -- 172.21.15.62:0/3550194814 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4053401547 0 0) 0x7fad040b4850 con 0x7fad0406d210 2024-04-18T10:07:00.925 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.921+0000 7fad02ffd640 1 -- 172.21.15.62:0/3550194814 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0409dc60 con 0x7fad040b3eb0 2024-04-18T10:07:00.926 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad02ffd640 1 --2- 172.21.15.62:0/3550194814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad040b3eb0 0x7fad040b4310 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7facf40097c0 tx=0x7facf401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56afc200ff4feb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad027fc640 1 -- 172.21.15.62:0/3550194814 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf40b0070 con 0x7fad040b3eb0 2024-04-18T10:07:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad027fc640 1 -- 172.21.15.62:0/3550194814 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facf40ab600 con 0x7fad040b3eb0 2024-04-18T10:07:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad027fc640 1 -- 172.21.15.62:0/3550194814 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf40ab9d0 con 0x7fad040b3eb0 2024-04-18T10:07:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad09c04640 1 -- 172.21.15.62:0/3550194814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad040b3eb0 msgr2=0x7fad040b4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad09c04640 1 --2- 172.21.15.62:0/3550194814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad040b3eb0 0x7fad040b4310 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7facf40097c0 tx=0x7facf401c380 comp rx=0 tx=0).stop 2024-04-18T10:07:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad09c04640 1 -- 172.21.15.62:0/3550194814 shutdown_connections 2024-04-18T10:07:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad09c04640 1 --2- 172.21.15.62:0/3550194814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad040b3eb0 0x7fad040b4310 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad09c04640 1 --2- 172.21.15.62:0/3550194814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0409c4d0 0x7fad0409c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad09c04640 1 -- 172.21.15.62:0/3550194814 >> 172.21.15.62:0/3550194814 conn(0x7fad040b9420 msgr2=0x7fad040b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad09c04640 1 -- 172.21.15.62:0/3550194814 shutdown_connections 2024-04-18T10:07:00.928 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad09c04640 1 -- 172.21.15.62:0/3550194814 wait complete. 2024-04-18T10:07:00.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad09c04640 1 Processor -- start 2024-04-18T10:07:00.930 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.925+0000 7fad09c04640 1 -- start start 2024-04-18T10:07:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad09c04640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0406d210 0x7fad04151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad09c04640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad0409c4d0 0x7fad041e3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad09c04640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad040b3eb0 0x7fad04099080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad09c04640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fad0406e2e0 con 0x7fad0409c4d0 2024-04-18T10:07:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad09c04640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fad0406e160 con 0x7fad040b3eb0 2024-04-18T10:07:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad09c04640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fad04155490 con 0x7fad0406d210 2024-04-18T10:07:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad02ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad0409c4d0 0x7fad041e3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad02ffd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad0409c4d0 0x7fad041e3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49150/0 (socket says 172.21.15.62:49150) 2024-04-18T10:07:00.931 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad02ffd640 1 -- 172.21.15.62:0/521330610 learned_addr learned my addr 172.21.15.62:0/521330610 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad037fe640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0406d210 0x7fad04151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad03fff640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad040b3eb0 0x7fad04099080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad02ffd640 1 -- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0406d210 msgr2=0x7fad04151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad02ffd640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0406d210 0x7fad04151d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad02ffd640 1 -- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad040b3eb0 msgr2=0x7fad04099080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad02ffd640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad040b3eb0 0x7fad04099080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad02ffd640 1 -- 172.21.15.62:0/521330610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0414d270 con 0x7fad0409c4d0 2024-04-18T10:07:00.932 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad037fe640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0406d210 0x7fad04151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad03fff640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad040b3eb0 0x7fad04099080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad02ffd640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad0409c4d0 0x7fad041e3bf0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7facf40ae040 tx=0x7facf4019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7facebfff640 1 -- 172.21.15.62:0/521330610 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf40b8030 con 0x7fad0409c4d0 2024-04-18T10:07:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7facebfff640 1 -- 172.21.15.62:0/521330610 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facf40ab400 con 0x7fad0409c4d0 2024-04-18T10:07:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad09c04640 1 -- 172.21.15.62:0/521330610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0414c340 con 0x7fad0409c4d0 2024-04-18T10:07:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7facebfff640 1 -- 172.21.15.62:0/521330610 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facf40b0040 con 0x7fad0409c4d0 2024-04-18T10:07:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad09c04640 1 -- 172.21.15.62:0/521330610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad0409b1b0 con 0x7fad0409c4d0 2024-04-18T10:07:00.933 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7fad09c04640 1 -- 172.21.15.62:0/521330610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad04068b20 con 0x7fad0409c4d0 2024-04-18T10:07:00.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7facebfff640 1 -- 172.21.15.62:0/521330610 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7facf40bb070 con 0x7fad0409c4d0 2024-04-18T10:07:00.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7facebfff640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7facf0068380 0x7facf006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:00.934 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.929+0000 7facebfff640 1 -- 172.21.15.62:0/521330610 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7facf411ff80 con 0x7fad0409c4d0 2024-04-18T10:07:00.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.933+0000 7fad037fe640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7facf0068380 0x7facf006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:00.935 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.933+0000 7fad037fe640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7facf0068380 0x7facf006a840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7facf8009840 tx=0x7facf8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:00.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:00.937+0000 7facebfff640 1 -- 172.21.15.62:0/521330610 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7facf40c2070 con 0x7fad0409c4d0 2024-04-18T10:07:01.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.097+0000 7fad09c04640 1 -- 172.21.15.62:0/521330610 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fad040b8650 con 0x7facf0068380 2024-04-18T10:07:01.099 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.097+0000 7facebfff640 1 -- 172.21.15.62:0/521330610 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fad040b8650 con 0x7facf0068380 2024-04-18T10:07:01.099 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 1 to deep-scrub 2024-04-18T10:07:01.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.101+0000 7fad09c04640 1 -- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7facf0068380 msgr2=0x7facf006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.101+0000 7fad09c04640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7facf0068380 0x7facf006a840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7facf8009840 tx=0x7facf8008040 comp rx=0 tx=0).stop 2024-04-18T10:07:01.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.101+0000 7fad09c04640 1 -- 172.21.15.62:0/521330610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad0409c4d0 msgr2=0x7fad041e3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.101+0000 7fad09c04640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad0409c4d0 0x7fad041e3bf0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7facf40ae040 tx=0x7facf4019120 comp rx=0 tx=0).stop 2024-04-18T10:07:01.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.101+0000 7fad09c04640 1 -- 172.21.15.62:0/521330610 shutdown_connections 2024-04-18T10:07:01.102 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.101+0000 7fad09c04640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7facf0068380 0x7facf006a840 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.101+0000 7fad09c04640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad040b3eb0 0x7fad04099080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.101+0000 7fad09c04640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fad0409c4d0 0x7fad041e3bf0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.101+0000 7fad09c04640 1 --2- 172.21.15.62:0/521330610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad0406d210 0x7fad04151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.101+0000 7fad09c04640 1 -- 172.21.15.62:0/521330610 >> 172.21.15.62:0/521330610 conn(0x7fad040b9420 msgr2=0x7fad040b9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:01.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.101+0000 7fad09c04640 1 -- 172.21.15.62:0/521330610 shutdown_connections 2024-04-18T10:07:01.103 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.101+0000 7fad09c04640 1 -- 172.21.15.62:0/521330610 wait complete. 2024-04-18T10:07:01.119 INFO:tasks.ceph:Scrubbing osd.2 2024-04-18T10:07:01.119 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-18T10:07:01.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.249+0000 7faaf64ba640 1 Processor -- start 2024-04-18T10:07:01.251 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.249+0000 7faaf64ba640 1 -- start start 2024-04-18T10:07:01.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faaf00b3e20 0x7faaf01461e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faaf006d920 0x7faaf0150470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaf00ba170 con 0x7faaf00b9d90 2024-04-18T10:07:01.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7faaf009cca0 con 0x7faaf006d920 2024-04-18T10:07:01.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7faaf0149930 con 0x7faaf00b3e20 2024-04-18T10:07:01.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf4a57640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faaf006d920 0x7faaf0150470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.253 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf4a57640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7faaf006d920 0x7faaf0150470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35690/0 (socket says 172.21.15.62:35690) 2024-04-18T10:07:01.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf4a57640 1 -- 172.21.15.62:0/1145817642 learned_addr learned my addr 172.21.15.62:0/1145817642 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:01.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaef7fe640 1 --2- 172.21.15.62:0/1145817642 >> v2:172.21.15.145:3300/0 conn(0x7faaf00b3e20 0x7faaf01461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf4a57640 1 -- 172.21.15.62:0/1145817642 >> v2:172.21.15.145:3300/0 conn(0x7faaf00b3e20 msgr2=0x7faaf01461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:01.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf4a57640 1 --2- 172.21.15.62:0/1145817642 >> v2:172.21.15.145:3300/0 conn(0x7faaf00b3e20 0x7faaf01461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf4a57640 1 -- 172.21.15.62:0/1145817642 >> v1:172.21.15.62:6789/0 conn(0x7faaf00b9d90 legacy=0x7faaf006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaeeffd640 1 -- 172.21.15.62:0/1145817642 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3736101829 0 0) 0x7faaf00ba170 con 0x7faaf00b9d90 2024-04-18T10:07:01.254 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf4a57640 1 -- 172.21.15.62:0/1145817642 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf0149c10 con 0x7faaf006d920 2024-04-18T10:07:01.255 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf4a57640 1 --2- 172.21.15.62:0/1145817642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaf006d920 0x7faaf0150470 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7faaf009b560 tx=0x7faae801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9231ca623cf7497a server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaeeffd640 1 -- 172.21.15.62:0/1145817642 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae80a5620 con 0x7faaf006d920 2024-04-18T10:07:01.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaeeffd640 1 -- 172.21.15.62:0/1145817642 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faae80a57c0 con 0x7faaf006d920 2024-04-18T10:07:01.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaeeffd640 1 -- 172.21.15.62:0/1145817642 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae80a5b90 con 0x7faaf006d920 2024-04-18T10:07:01.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 -- 172.21.15.62:0/1145817642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaf006d920 msgr2=0x7faaf0150470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 --2- 172.21.15.62:0/1145817642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaf006d920 0x7faaf0150470 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7faaf009b560 tx=0x7faae801c960 comp rx=0 tx=0).stop 2024-04-18T10:07:01.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 -- 172.21.15.62:0/1145817642 shutdown_connections 2024-04-18T10:07:01.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 --2- 172.21.15.62:0/1145817642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaf006d920 0x7faaf0150470 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.256 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 --2- 172.21.15.62:0/1145817642 >> v2:172.21.15.145:3300/0 conn(0x7faaf00b3e20 0x7faaf01461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 -- 172.21.15.62:0/1145817642 >> 172.21.15.62:0/1145817642 conn(0x7faaf006c0d0 msgr2=0x7faaf00b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:01.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 -- 172.21.15.62:0/1145817642 shutdown_connections 2024-04-18T10:07:01.257 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.253+0000 7faaf64ba640 1 -- 172.21.15.62:0/1145817642 wait complete. 2024-04-18T10:07:01.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaf64ba640 1 Processor -- start 2024-04-18T10:07:01.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaf64ba640 1 -- start start 2024-04-18T10:07:01.259 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaf64ba640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaf006d920 0x7faaf013d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaf64ba640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faaf00b3e20 0x7faaf00b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaf64ba640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faaf00b9d90 0x7faaf0098460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaf64ba640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7faaf01533e0 con 0x7faaf006d920 2024-04-18T10:07:01.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaf64ba640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7faaf009c020 con 0x7faaf00b3e20 2024-04-18T10:07:01.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaf64ba640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7faaf0153560 con 0x7faaf00b9d90 2024-04-18T10:07:01.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaeffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaf006d920 0x7faaf013d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaeffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaf006d920 0x7faaf013d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35698/0 (socket says 172.21.15.62:35698) 2024-04-18T10:07:01.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaeffff640 1 -- 172.21.15.62:0/2108160500 learned_addr learned my addr 172.21.15.62:0/2108160500 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:01.260 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaef7fe640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faaf00b3e20 0x7faaf00b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaf4a57640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faaf00b9d90 0x7faaf0098460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaeffff640 1 -- 172.21.15.62:0/2108160500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faaf00b9d90 msgr2=0x7faaf0098460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaeffff640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faaf00b9d90 0x7faaf0098460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaeffff640 1 -- 172.21.15.62:0/2108160500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faaf00b3e20 msgr2=0x7faaf00b7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaeffff640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faaf00b3e20 0x7faaf00b7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaeffff640 1 -- 172.21.15.62:0/2108160500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf0099f80 con 0x7faaf006d920 2024-04-18T10:07:01.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaeffff640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaf006d920 0x7faaf013d970 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7faad800c500 tx=0x7faad800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaecff9640 1 -- 172.21.15.62:0/2108160500 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8017070 con 0x7faaf006d920 2024-04-18T10:07:01.261 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaecff9640 1 -- 172.21.15.62:0/2108160500 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faad8003600 con 0x7faaf006d920 2024-04-18T10:07:01.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaf64ba640 1 -- 172.21.15.62:0/2108160500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf009a260 con 0x7faaf006d920 2024-04-18T10:07:01.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaecff9640 1 -- 172.21.15.62:0/2108160500 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad80039f0 con 0x7faaf006d920 2024-04-18T10:07:01.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaf64ba640 1 -- 172.21.15.62:0/2108160500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf009aa60 con 0x7faaf006d920 2024-04-18T10:07:01.262 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.257+0000 7faaf64ba640 1 -- 172.21.15.62:0/2108160500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faaf009a650 con 0x7faaf006d920 2024-04-18T10:07:01.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.261+0000 7faaecff9640 1 -- 172.21.15.62:0/2108160500 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faad801a070 con 0x7faaf006d920 2024-04-18T10:07:01.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.261+0000 7faaecff9640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faad0068380 0x7faad006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.263 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.261+0000 7faaecff9640 1 -- 172.21.15.62:0/2108160500 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7faad80861d0 con 0x7faaf006d920 2024-04-18T10:07:01.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.261+0000 7faaecff9640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] conn(0x7faad006e240 0x7faad0070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.261+0000 7faaecff9640 1 -- 172.21.15.62:0/2108160500 --> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faad0070d10 con 0x7faad006e240 2024-04-18T10:07:01.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.261+0000 7faaecff9640 1 -- 172.21.15.62:0/2108160500 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7faad8086660 con 0x7faaf006d920 2024-04-18T10:07:01.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.261+0000 7faaef7fe640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faad0068380 0x7faad006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.261+0000 7faaf4a57640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] conn(0x7faad006e240 0x7faad0070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.261+0000 7faaf4a57640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] conn(0x7faad006e240 0x7faad0070660 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.261+0000 7faaef7fe640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faad0068380 0x7faad006a840 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7faaf01e4950 tx=0x7faae0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.264 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.261+0000 7faaecff9640 1 -- 172.21.15.62:0/2108160500 <== osd.2 v2:172.21.15.62:6816/1732263675 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faad0070d10 con 0x7faad006e240 2024-04-18T10:07:01.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaf64ba640 1 -- 172.21.15.62:0/2108160500 --> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7faaf0068ca0 con 0x7faad006e240 2024-04-18T10:07:01.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaecff9640 1 -- 172.21.15.62:0/2108160500 <== osd.2 v2:172.21.15.62:6816/1732263675 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7faaf0068ca0 con 0x7faad006e240 2024-04-18T10:07:01.275 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:07:01.275 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T10:07:01.275 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:07:01.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaf64ba640 1 -- 172.21.15.62:0/2108160500 >> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] conn(0x7faad006e240 msgr2=0x7faad0070660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaf64ba640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.62:6816/1732263675,v1:172.21.15.62:6818/1732263675] conn(0x7faad006e240 0x7faad0070660 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaf64ba640 1 -- 172.21.15.62:0/2108160500 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faad0068380 msgr2=0x7faad006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaf64ba640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7faad0068380 0x7faad006a840 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7faaf01e4950 tx=0x7faae0008040 comp rx=0 tx=0).stop 2024-04-18T10:07:01.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaf64ba640 1 -- 172.21.15.62:0/2108160500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaf006d920 msgr2=0x7faaf013d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaf64ba640 1 --2- 172.21.15.62:0/2108160500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaf006d920 0x7faaf013d970 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7faad800c500 tx=0x7faad800c9c0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaf4a57640 1 -- 172.21.15.62:0/2108160500 reap_dead start 2024-04-18T10:07:01.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaf64ba640 1 -- 172.21.15.62:0/2108160500 shutdown_connections 2024-04-18T10:07:01.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaf64ba640 1 -- 172.21.15.62:0/2108160500 >> 172.21.15.62:0/2108160500 conn(0x7faaf006c0d0 msgr2=0x7faaf00aeca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:01.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaf64ba640 1 -- 172.21.15.62:0/2108160500 shutdown_connections 2024-04-18T10:07:01.278 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.273+0000 7faaf64ba640 1 -- 172.21.15.62:0/2108160500 wait complete. 2024-04-18T10:07:01.291 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-18T10:07:01.395 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.393+0000 7fce4b5f3640 1 Processor -- start 2024-04-18T10:07:01.396 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.393+0000 7fce4b5f3640 1 -- start start 2024-04-18T10:07:01.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce4b5f3640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fce440b6300 0x7fce4414b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce4b5f3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce440b74e0 con 0x7fce44148240 2024-04-18T10:07:01.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce4b5f3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce44147d90 con 0x7fce441486d0 2024-04-18T10:07:01.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce4b5f3640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fce440b7750 con 0x7fce440b6300 2024-04-18T10:07:01.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce48b8e640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fce44148240 0x7fce44141770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53978/0 (socket says 172.21.15.62:53978) 2024-04-18T10:07:01.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce48b8e640 1 -- 172.21.15.62:0/3341336908 learned_addr learned my addr 172.21.15.62:0/3341336908 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:07:01.399 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1021873903 0 0) 0x7fce440b74e0 con 0x7fce44148240 2024-04-18T10:07:01.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce30003530 con 0x7fce44148240 2024-04-18T10:07:01.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce4938f640 1 --2- 172.21.15.62:0/3341336908 >> v2:172.21.15.145:3300/0 conn(0x7fce440b6300 0x7fce4414b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1152698182 0 0) 0x7fce30003530 con 0x7fce44148240 2024-04-18T10:07:01.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 >> v2:172.21.15.145:3300/0 conn(0x7fce440b6300 msgr2=0x7fce4414b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:01.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 --2- 172.21.15.62:0/3341336908 >> v2:172.21.15.145:3300/0 conn(0x7fce440b6300 0x7fce4414b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 >> v1:172.21.15.145:6789/0 conn(0x7fce441486d0 legacy=0x7fce44152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:01.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce440b66e0 con 0x7fce44148240 2024-04-18T10:07:01.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fce38001a00 con 0x7fce44148240 2024-04-18T10:07:01.400 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 >> v1:172.21.15.62:6789/0 conn(0x7fce44148240 legacy=0x7fce44141770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 --2- 172.21.15.62:0/3341336908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300039c0 0x7fce30003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 --2- 172.21.15.62:0/3341336908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce30004810 0x7fce30004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 --2- 172.21.15.62:0/3341336908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce300051d0 0x7fce3000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fce440b7750 con 0x7fce300039c0 2024-04-18T10:07:01.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fce3000c100 con 0x7fce300051d0 2024-04-18T10:07:01.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fce3000c300 con 0x7fce30004810 2024-04-18T10:07:01.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce4938f640 1 --2- 172.21.15.62:0/3341336908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300039c0 0x7fce30003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce48b8e640 1 --2- 172.21.15.62:0/3341336908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce30004810 0x7fce30004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.401 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce4938f640 1 -- 172.21.15.62:0/3341336908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce30004810 msgr2=0x7fce30004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:01.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce4938f640 1 --2- 172.21.15.62:0/3341336908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce30004810 0x7fce30004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce4938f640 1 -- 172.21.15.62:0/3341336908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce300051d0 msgr2=0x7fce3000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:01.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce49b90640 1 --2- 172.21.15.62:0/3341336908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce300051d0 0x7fce3000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce4938f640 1 --2- 172.21.15.62:0/3341336908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce300051d0 0x7fce3000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce4938f640 1 -- 172.21.15.62:0/3341336908 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce30003530 con 0x7fce300039c0 2024-04-18T10:07:01.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce49b90640 1 --2- 172.21.15.62:0/3341336908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce300051d0 0x7fce3000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:01.402 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce49b90640 1 -- 172.21.15.62:0/3341336908 reap_dead start 2024-04-18T10:07:01.405 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce4938f640 1 --2- 172.21.15.62:0/3341336908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300039c0 0x7fce30003dc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fce4000ba00 tx=0x7fce4001c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cbd0004a5a60695 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.397+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce400b2070 con 0x7fce300039c0 2024-04-18T10:07:01.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce400ad630 con 0x7fce300039c0 2024-04-18T10:07:01.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce37fff640 1 -- 172.21.15.62:0/3341336908 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4000b600 con 0x7fce300039c0 2024-04-18T10:07:01.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 -- 172.21.15.62:0/3341336908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300039c0 msgr2=0x7fce30003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 --2- 172.21.15.62:0/3341336908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300039c0 0x7fce30003dc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fce4000ba00 tx=0x7fce4001c630 comp rx=0 tx=0).stop 2024-04-18T10:07:01.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 -- 172.21.15.62:0/3341336908 shutdown_connections 2024-04-18T10:07:01.406 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 --2- 172.21.15.62:0/3341336908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce300039c0 0x7fce30003dc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 -- 172.21.15.62:0/3341336908 >> 172.21.15.62:0/3341336908 conn(0x7fce44068090 msgr2=0x7fce440bd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:01.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 -- 172.21.15.62:0/3341336908 shutdown_connections 2024-04-18T10:07:01.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 -- 172.21.15.62:0/3341336908 wait complete. 2024-04-18T10:07:01.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 Processor -- start 2024-04-18T10:07:01.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 -- start start 2024-04-18T10:07:01.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300039c0 0x7fce441d3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce441e11d0 0x7fce441e1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.407 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce4413d6c0 0x7fce4413db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fce44142100 con 0x7fce4413d6c0 2024-04-18T10:07:01.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fce44141f80 con 0x7fce441e11d0 2024-04-18T10:07:01.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fce44142280 con 0x7fce300039c0 2024-04-18T10:07:01.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce49b90640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce4413d6c0 0x7fce4413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce49b90640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce4413d6c0 0x7fce4413db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35724/0 (socket says 172.21.15.62:35724) 2024-04-18T10:07:01.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce49b90640 1 -- 172.21.15.62:0/1379240360 learned_addr learned my addr 172.21.15.62:0/1379240360 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:01.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4938f640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300039c0 0x7fce441d3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce48b8e640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce441e11d0 0x7fce441e1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.408 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce49b90640 1 -- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300039c0 msgr2=0x7fce441d3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce49b90640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300039c0 0x7fce441d3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce49b90640 1 -- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce441e11d0 msgr2=0x7fce441e1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce49b90640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce441e11d0 0x7fce441e1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce49b90640 1 -- 172.21.15.62:0/1379240360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce441d6520 con 0x7fce4413d6c0 2024-04-18T10:07:01.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce48b8e640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce441e11d0 0x7fce441e1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:01.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce49b90640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce4413d6c0 0x7fce4413db40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fce4406c760 tx=0x7fce3c00d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce35ffb640 1 -- 172.21.15.62:0/1379240360 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c005230 con 0x7fce4413d6c0 2024-04-18T10:07:01.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce35ffb640 1 -- 172.21.15.62:0/1379240360 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce3c0053f0 con 0x7fce4413d6c0 2024-04-18T10:07:01.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 -- 172.21.15.62:0/1379240360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce441e1b90 con 0x7fce4413d6c0 2024-04-18T10:07:01.409 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce35ffb640 1 -- 172.21.15.62:0/1379240360 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c007260 con 0x7fce4413d6c0 2024-04-18T10:07:01.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.401+0000 7fce4b5f3640 1 -- 172.21.15.62:0/1379240360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce441d52a0 con 0x7fce4413d6c0 2024-04-18T10:07:01.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.405+0000 7fce4b5f3640 1 -- 172.21.15.62:0/1379240360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce44068b20 con 0x7fce4413d6c0 2024-04-18T10:07:01.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.405+0000 7fce35ffb640 1 -- 172.21.15.62:0/1379240360 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fce3c007400 con 0x7fce4413d6c0 2024-04-18T10:07:01.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.405+0000 7fce35ffb640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fce28068380 0x7fce2806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.405+0000 7fce35ffb640 1 -- 172.21.15.62:0/1379240360 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fce3c085820 con 0x7fce4413d6c0 2024-04-18T10:07:01.410 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.405+0000 7fce4938f640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fce28068380 0x7fce2806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.405+0000 7fce4938f640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fce28068380 0x7fce2806a840 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fce400af000 tx=0x7fce400046d0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.411 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.409+0000 7fce35ffb640 1 -- 172.21.15.62:0/1379240360 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fce441e1b90 con 0x7fce4413d6c0 2024-04-18T10:07:01.564 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.565+0000 7fce4b5f3640 1 -- 172.21.15.62:0/1379240360 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fce4414b510 con 0x7fce28068380 2024-04-18T10:07:01.565 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.565+0000 7fce35ffb640 1 -- 172.21.15.62:0/1379240360 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fce4414b510 con 0x7fce28068380 2024-04-18T10:07:01.565 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 2 to deep-scrub 2024-04-18T10:07:01.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.565+0000 7fce4b5f3640 1 -- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fce28068380 msgr2=0x7fce2806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.565+0000 7fce4b5f3640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fce28068380 0x7fce2806a840 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fce400af000 tx=0x7fce400046d0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.565+0000 7fce4b5f3640 1 -- 172.21.15.62:0/1379240360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce4413d6c0 msgr2=0x7fce4413db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.565+0000 7fce4b5f3640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce4413d6c0 0x7fce4413db40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fce4406c760 tx=0x7fce3c00d890 comp rx=0 tx=0).stop 2024-04-18T10:07:01.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.565+0000 7fce4b5f3640 1 -- 172.21.15.62:0/1379240360 shutdown_connections 2024-04-18T10:07:01.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.565+0000 7fce4b5f3640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fce28068380 0x7fce2806a840 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.565+0000 7fce4b5f3640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fce4413d6c0 0x7fce4413db40 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.565+0000 7fce4b5f3640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce441e11d0 0x7fce441e1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.565+0000 7fce4b5f3640 1 --2- 172.21.15.62:0/1379240360 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300039c0 0x7fce441d3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.565+0000 7fce4b5f3640 1 -- 172.21.15.62:0/1379240360 >> 172.21.15.62:0/1379240360 conn(0x7fce44068090 msgr2=0x7fce44145bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:01.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.569+0000 7fce4b5f3640 1 -- 172.21.15.62:0/1379240360 shutdown_connections 2024-04-18T10:07:01.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.569+0000 7fce4b5f3640 1 -- 172.21.15.62:0/1379240360 wait complete. 2024-04-18T10:07:01.583 INFO:tasks.ceph:Scrubbing osd.3 2024-04-18T10:07:01.583 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-18T10:07:01.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d37b03640 1 Processor -- start 2024-04-18T10:07:01.701 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d37b03640 1 -- start start 2024-04-18T10:07:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d37b03640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d300b9d90 0x7f7d300b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d37b03640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7d301480e0 0x7f7d30148560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d37b03640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7d30143f00 0x7f7d30144390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d37b03640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f7d300b5b00 con 0x7f7d301480e0 2024-04-18T10:07:01.702 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d37b03640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f7d300ba170 con 0x7f7d30143f00 2024-04-18T10:07:01.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d37b03640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7d3009dc10 con 0x7f7d300b9d90 2024-04-18T10:07:01.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d3509e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7d301480e0 0x7f7d30148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d3509e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f7d301480e0 0x7f7d30148560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35726/0 (socket says 172.21.15.62:35726) 2024-04-18T10:07:01.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d3509e640 1 -- 172.21.15.62:0/1641736025 learned_addr learned my addr 172.21.15.62:0/1641736025 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:01.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d3509e640 1 -- 172.21.15.62:0/1641736025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d300b9d90 msgr2=0x7f7d300b55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:01.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d3509e640 1 --2- 172.21.15.62:0/1641736025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d300b9d90 0x7f7d300b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d3509e640 1 -- 172.21.15.62:0/1641736025 >> v2:172.21.15.145:3300/0 conn(0x7f7d30143f00 msgr2=0x7f7d30144390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:01.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d3509e640 1 --2- 172.21.15.62:0/1641736025 >> v2:172.21.15.145:3300/0 conn(0x7f7d30143f00 0x7f7d30144390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.701+0000 7f7d3509e640 1 -- 172.21.15.62:0/1641736025 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d3009d410 con 0x7f7d301480e0 2024-04-18T10:07:01.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d3509e640 1 --2- 172.21.15.62:0/1641736025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301480e0 0x7f7d30148560 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7d300b5000 tx=0x7f7d2c01ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61deb14b3f51f31 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d3489d640 1 -- 172.21.15.62:0/1641736025 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c0b4070 con 0x7f7d301480e0 2024-04-18T10:07:01.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d3489d640 1 -- 172.21.15.62:0/1641736025 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d2c0a7960 con 0x7f7d301480e0 2024-04-18T10:07:01.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d3489d640 1 -- 172.21.15.62:0/1641736025 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c0a7d30 con 0x7f7d301480e0 2024-04-18T10:07:01.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 -- 172.21.15.62:0/1641736025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301480e0 msgr2=0x7f7d30148560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 --2- 172.21.15.62:0/1641736025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301480e0 0x7f7d30148560 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7d300b5000 tx=0x7f7d2c01ead0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 -- 172.21.15.62:0/1641736025 shutdown_connections 2024-04-18T10:07:01.705 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 --2- 172.21.15.62:0/1641736025 >> v2:172.21.15.145:3300/0 conn(0x7f7d30143f00 0x7f7d30144390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 --2- 172.21.15.62:0/1641736025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301480e0 0x7f7d30148560 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 --2- 172.21.15.62:0/1641736025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d300b9d90 0x7f7d300b55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 -- 172.21.15.62:0/1641736025 >> 172.21.15.62:0/1641736025 conn(0x7f7d3006c0d0 msgr2=0x7f7d300b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:01.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 -- 172.21.15.62:0/1641736025 shutdown_connections 2024-04-18T10:07:01.706 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 -- 172.21.15.62:0/1641736025 wait complete. 2024-04-18T10:07:01.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 Processor -- start 2024-04-18T10:07:01.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 -- start start 2024-04-18T10:07:01.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d300b9d90 0x7f7d300b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.708 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d30143f00 0x7f7d30072b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301480e0 0x7f7d30142050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7d300b47b0 con 0x7f7d301480e0 2024-04-18T10:07:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7d300b4630 con 0x7f7d30143f00 2024-04-18T10:07:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7d300b4930 con 0x7f7d300b9d90 2024-04-18T10:07:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d360a0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301480e0 0x7f7d30142050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d360a0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301480e0 0x7f7d30142050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35730/0 (socket says 172.21.15.62:35730) 2024-04-18T10:07:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d360a0640 1 -- 172.21.15.62:0/1327375709 learned_addr learned my addr 172.21.15.62:0/1327375709 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:01.709 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d3589f640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d300b9d90 0x7f7d300b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d3509e640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d30143f00 0x7f7d30072b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d360a0640 1 -- 172.21.15.62:0/1327375709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d300b9d90 msgr2=0x7f7d300b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d360a0640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d300b9d90 0x7f7d300b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d360a0640 1 -- 172.21.15.62:0/1327375709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d30143f00 msgr2=0x7f7d30072b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d360a0640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d30143f00 0x7f7d30072b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d360a0640 1 -- 172.21.15.62:0/1327375709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d30072690 con 0x7f7d301480e0 2024-04-18T10:07:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d3589f640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d300b9d90 0x7f7d300b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d360a0640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301480e0 0x7f7d30142050 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800b9a0 tx=0x7f7d2800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.710 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d267fc640 1 -- 172.21.15.62:0/1327375709 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28019070 con 0x7f7d301480e0 2024-04-18T10:07:01.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d267fc640 1 -- 172.21.15.62:0/1327375709 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d2800cbc0 con 0x7f7d301480e0 2024-04-18T10:07:01.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 -- 172.21.15.62:0/1327375709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d30073080 con 0x7f7d301480e0 2024-04-18T10:07:01.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d267fc640 1 -- 172.21.15.62:0/1327375709 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d28014440 con 0x7f7d301480e0 2024-04-18T10:07:01.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.705+0000 7f7d37b03640 1 -- 172.21.15.62:0/1327375709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d30072200 con 0x7f7d301480e0 2024-04-18T10:07:01.711 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.709+0000 7f7d37b03640 1 -- 172.21.15.62:0/1327375709 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7d30068ca0 con 0x7f7d301480e0 2024-04-18T10:07:01.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.709+0000 7f7d267fc640 1 -- 172.21.15.62:0/1327375709 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7d2801c070 con 0x7f7d301480e0 2024-04-18T10:07:01.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.709+0000 7f7d267fc640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7d14068380 0x7f7d1406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.712 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.709+0000 7f7d267fc640 1 -- 172.21.15.62:0/1327375709 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f7d28086b90 con 0x7f7d301480e0 2024-04-18T10:07:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.709+0000 7f7d267fc640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] conn(0x7f7d1406e240 0x7f7d14070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.709+0000 7f7d267fc640 1 -- 172.21.15.62:0/1327375709 --> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d14070d10 con 0x7f7d1406e240 2024-04-18T10:07:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.709+0000 7f7d267fc640 1 -- 172.21.15.62:0/1327375709 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f7d28087020 con 0x7f7d301480e0 2024-04-18T10:07:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.709+0000 7f7d3589f640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7d14068380 0x7f7d1406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.709+0000 7f7d3509e640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] conn(0x7f7d1406e240 0x7f7d14070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.709+0000 7f7d3509e640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] conn(0x7f7d1406e240 0x7f7d14070660 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.709+0000 7f7d3589f640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7d14068380 0x7f7d1406a840 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d301410e0 tx=0x7f7d20008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.713 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.709+0000 7f7d267fc640 1 -- 172.21.15.62:0/1327375709 <== osd.3 v2:172.21.15.62:6806/1675715975 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7d14070d10 con 0x7f7d1406e240 2024-04-18T10:07:01.723 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.721+0000 7f7d37b03640 1 -- 172.21.15.62:0/1327375709 --> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7d301457f0 con 0x7f7d1406e240 2024-04-18T10:07:01.724 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.725+0000 7f7d267fc640 1 -- 172.21.15.62:0/1327375709 <== osd.3 v2:172.21.15.62:6806/1675715975 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7d301457f0 con 0x7f7d1406e240 2024-04-18T10:07:01.724 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:07:01.724 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T10:07:01.724 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:07:01.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.725+0000 7f7d37b03640 1 -- 172.21.15.62:0/1327375709 >> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] conn(0x7f7d1406e240 msgr2=0x7f7d14070660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.725+0000 7f7d37b03640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.62:6806/1675715975,v1:172.21.15.62:6809/1675715975] conn(0x7f7d1406e240 0x7f7d14070660 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.725 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.725+0000 7f7d37b03640 1 -- 172.21.15.62:0/1327375709 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7d14068380 msgr2=0x7f7d1406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.725+0000 7f7d37b03640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7d14068380 0x7f7d1406a840 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7d301410e0 tx=0x7f7d20008040 comp rx=0 tx=0).stop 2024-04-18T10:07:01.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.725+0000 7f7d37b03640 1 -- 172.21.15.62:0/1327375709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301480e0 msgr2=0x7f7d30142050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.725+0000 7f7d37b03640 1 --2- 172.21.15.62:0/1327375709 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301480e0 0x7f7d30142050 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7d2800b9a0 tx=0x7f7d2800be60 comp rx=0 tx=0).stop 2024-04-18T10:07:01.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.725+0000 7f7d360a0640 1 -- 172.21.15.62:0/1327375709 reap_dead start 2024-04-18T10:07:01.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.725+0000 7f7d37b03640 1 -- 172.21.15.62:0/1327375709 shutdown_connections 2024-04-18T10:07:01.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.725+0000 7f7d37b03640 1 -- 172.21.15.62:0/1327375709 >> 172.21.15.62:0/1327375709 conn(0x7f7d3006c0d0 msgr2=0x7f7d300736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:01.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.725+0000 7f7d37b03640 1 -- 172.21.15.62:0/1327375709 shutdown_connections 2024-04-18T10:07:01.726 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.725+0000 7f7d37b03640 1 -- 172.21.15.62:0/1327375709 wait complete. 2024-04-18T10:07:01.739 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-18T10:07:01.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f6216878640 1 Processor -- start 2024-04-18T10:07:01.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f6216878640 1 -- start start 2024-04-18T10:07:01.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f6216878640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f621009c8b0 0x7f6210153380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f6216878640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f621009dc60 con 0x7f62100b45c0 2024-04-18T10:07:01.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f6216878640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62100b5bc0 con 0x7f621009c4d0 2024-04-18T10:07:01.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f6216878640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f6210098da0 con 0x7f621009c8b0 2024-04-18T10:07:01.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f6214e15640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f621009c8b0 0x7f6210153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f620f7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f62100b45c0 0x7f621014fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53980/0 (socket says 172.21.15.62:53980) 2024-04-18T10:07:01.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f620f7fe640 1 -- 172.21.15.62:0/496546371 learned_addr learned my addr 172.21.15.62:0/496546371 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:07:01.850 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f6214e15640 1 --2- 172.21.15.62:0/496546371 >> v2:172.21.15.62:3300/0 conn(0x7f621009c8b0 0x7f6210153380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35740/0 (socket says 172.21.15.62:35740) 2024-04-18T10:07:01.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f6214e15640 1 -- 172.21.15.62:0/496546371 >> v1:172.21.15.145:6789/0 conn(0x7f621009c4d0 legacy=0x7f62100b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:01.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f620effd640 1 -- 172.21.15.62:0/496546371 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3417870451 0 0) 0x7f621009dc60 con 0x7f62100b45c0 2024-04-18T10:07:01.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f6214e15640 1 -- 172.21.15.62:0/496546371 >> v1:172.21.15.62:6789/0 conn(0x7f62100b45c0 legacy=0x7f621014fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.851 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.849+0000 7f6214e15640 1 -- 172.21.15.62:0/496546371 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6210073da0 con 0x7f621009c8b0 2024-04-18T10:07:01.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6214e15640 1 --2- 172.21.15.62:0/496546371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f621009c8b0 0x7f6210153380 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6210098f20 tx=0x7f620801c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689bb9056cb2ffa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620effd640 1 -- 172.21.15.62:0/496546371 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62080a6700 con 0x7f621009c8b0 2024-04-18T10:07:01.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620effd640 1 -- 172.21.15.62:0/496546371 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62080a68a0 con 0x7f621009c8b0 2024-04-18T10:07:01.852 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620effd640 1 -- 172.21.15.62:0/496546371 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62080a6c70 con 0x7f621009c8b0 2024-04-18T10:07:01.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 -- 172.21.15.62:0/496546371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f621009c8b0 msgr2=0x7f6210153380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 --2- 172.21.15.62:0/496546371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f621009c8b0 0x7f6210153380 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f6210098f20 tx=0x7f620801c830 comp rx=0 tx=0).stop 2024-04-18T10:07:01.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 -- 172.21.15.62:0/496546371 shutdown_connections 2024-04-18T10:07:01.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 --2- 172.21.15.62:0/496546371 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f621009c8b0 0x7f6210153380 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 -- 172.21.15.62:0/496546371 >> 172.21.15.62:0/496546371 conn(0x7f62100b9420 msgr2=0x7f62100b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:01.853 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 -- 172.21.15.62:0/496546371 shutdown_connections 2024-04-18T10:07:01.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 -- 172.21.15.62:0/496546371 wait complete. 2024-04-18T10:07:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 Processor -- start 2024-04-18T10:07:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 -- start start 2024-04-18T10:07:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f621009c4d0 0x7f62100b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f621009c8b0 0x7f6210145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62100b45c0 0x7f62100b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f6210156700 con 0x7f621009c8b0 2024-04-18T10:07:01.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f621009d980 con 0x7f62100b45c0 2024-04-18T10:07:01.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f6210156880 con 0x7f621009c4d0 2024-04-18T10:07:01.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f621009c8b0 0x7f6210145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620f7fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f621009c8b0 0x7f6210145ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35756/0 (socket says 172.21.15.62:35756) 2024-04-18T10:07:01.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620f7fe640 1 -- 172.21.15.62:0/1700679976 learned_addr learned my addr 172.21.15.62:0/1700679976 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:01.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620ffff640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f621009c4d0 0x7f62100b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6214e15640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62100b45c0 0x7f62100b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620f7fe640 1 -- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f621009c4d0 msgr2=0x7f62100b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620f7fe640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f621009c4d0 0x7f62100b1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.857 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620f7fe640 1 -- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62100b45c0 msgr2=0x7f62100b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:01.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620f7fe640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62100b45c0 0x7f62100b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:01.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620f7fe640 1 -- 172.21.15.62:0/1700679976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62100b25c0 con 0x7f621009c8b0 2024-04-18T10:07:01.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620ffff640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f621009c4d0 0x7f62100b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:01.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620f7fe640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f621009c8b0 0x7f6210145ff0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00c4a0 tx=0x7f61fc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620cff9640 1 -- 172.21.15.62:0/1700679976 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc018070 con 0x7f621009c8b0 2024-04-18T10:07:01.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620cff9640 1 -- 172.21.15.62:0/1700679976 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61fc0074e0 con 0x7f621009c8b0 2024-04-18T10:07:01.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f620cff9640 1 -- 172.21.15.62:0/1700679976 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61fc004a30 con 0x7f621009c8b0 2024-04-18T10:07:01.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 -- 172.21.15.62:0/1700679976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6210071f70 con 0x7f621009c8b0 2024-04-18T10:07:01.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.853+0000 7f6216878640 1 -- 172.21.15.62:0/1700679976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6210072770 con 0x7f621009c8b0 2024-04-18T10:07:01.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.857+0000 7f6216878640 1 -- 172.21.15.62:0/1700679976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6210068b20 con 0x7f621009c8b0 2024-04-18T10:07:01.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.857+0000 7f620cff9640 1 -- 172.21.15.62:0/1700679976 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f61fc01b070 con 0x7f621009c8b0 2024-04-18T10:07:01.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.857+0000 7f620cff9640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f61f8068380 0x7f61f806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:01.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.857+0000 7f620cff9640 1 -- 172.21.15.62:0/1700679976 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f61fc0858f0 con 0x7f621009c8b0 2024-04-18T10:07:01.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.857+0000 7f620ffff640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f61f8068380 0x7f61f806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:01.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.857+0000 7f620ffff640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f61f8068380 0x7f61f806a840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6200000f70 tx=0x7f6200009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:01.861 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:01.861+0000 7f620cff9640 1 -- 172.21.15.62:0/1700679976 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f61fc056240 con 0x7f621009c8b0 2024-04-18T10:07:02.019 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.017+0000 7f6216878640 1 -- 172.21.15.62:0/1700679976 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f62100008e0 con 0x7f61f8068380 2024-04-18T10:07:02.020 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.017+0000 7f620cff9640 1 -- 172.21.15.62:0/1700679976 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f62100008e0 con 0x7f61f8068380 2024-04-18T10:07:02.020 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 3 to deep-scrub 2024-04-18T10:07:02.021 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.021+0000 7f6216878640 1 -- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f61f8068380 msgr2=0x7f61f806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.021+0000 7f6216878640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f61f8068380 0x7f61f806a840 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6200000f70 tx=0x7f6200009040 comp rx=0 tx=0).stop 2024-04-18T10:07:02.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.021+0000 7f6216878640 1 -- 172.21.15.62:0/1700679976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f621009c8b0 msgr2=0x7f6210145ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.021+0000 7f6216878640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f621009c8b0 0x7f6210145ff0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f61fc00c4a0 tx=0x7f61fc00c960 comp rx=0 tx=0).stop 2024-04-18T10:07:02.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.021+0000 7f6216878640 1 -- 172.21.15.62:0/1700679976 shutdown_connections 2024-04-18T10:07:02.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.021+0000 7f6216878640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f61f8068380 0x7f61f806a840 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.021+0000 7f6216878640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62100b45c0 0x7f62100b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.022 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.021+0000 7f6216878640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f621009c8b0 0x7f6210145ff0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.021+0000 7f6216878640 1 --2- 172.21.15.62:0/1700679976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f621009c4d0 0x7f62100b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.021+0000 7f6216878640 1 -- 172.21.15.62:0/1700679976 >> 172.21.15.62:0/1700679976 conn(0x7f62100b9420 msgr2=0x7f62100b5eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:02.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.021+0000 7f6216878640 1 -- 172.21.15.62:0/1700679976 shutdown_connections 2024-04-18T10:07:02.023 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.021+0000 7f6216878640 1 -- 172.21.15.62:0/1700679976 wait complete. 2024-04-18T10:07:02.037 INFO:tasks.ceph:Scrubbing osd.4 2024-04-18T10:07:02.037 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-18T10:07:02.160 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff444251640 1 Processor -- start 2024-04-18T10:07:02.161 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff444251640 1 -- start start 2024-04-18T10:07:02.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff444251640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff43c0b9d90 0x7ff43c0b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff444251640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff43c1480e0 0x7ff43c148560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.162 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff444251640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff43c143f00 0x7ff43c144390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff444251640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7ff43c0b5b00 con 0x7ff43c0b9d90 2024-04-18T10:07:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff444251640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7ff43c0ba170 con 0x7ff43c143f00 2024-04-18T10:07:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff444251640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7ff43c09dc10 con 0x7ff43c1480e0 2024-04-18T10:07:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff441fed640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff43c0b9d90 0x7ff43c0b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff441fed640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7ff43c0b9d90 0x7ff43c0b55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35760/0 (socket says 172.21.15.62:35760) 2024-04-18T10:07:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff441fed640 1 -- 172.21.15.62:0/3881112653 learned_addr learned my addr 172.21.15.62:0/3881112653 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff4417ec640 1 --2- 172.21.15.62:0/3881112653 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff43c1480e0 0x7ff43c148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.163 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff4427ee640 1 --2- 172.21.15.62:0/3881112653 >> v2:172.21.15.145:3300/0 conn(0x7ff43c143f00 0x7ff43c144390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff441fed640 1 -- 172.21.15.62:0/3881112653 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff43c1480e0 msgr2=0x7ff43c148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff441fed640 1 --2- 172.21.15.62:0/3881112653 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff43c1480e0 0x7ff43c148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff441fed640 1 -- 172.21.15.62:0/3881112653 >> v2:172.21.15.145:3300/0 conn(0x7ff43c143f00 msgr2=0x7ff43c144390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff441fed640 1 --2- 172.21.15.62:0/3881112653 >> v2:172.21.15.145:3300/0 conn(0x7ff43c143f00 0x7ff43c144390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff441fed640 1 -- 172.21.15.62:0/3881112653 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43c09d410 con 0x7ff43c0b9d90 2024-04-18T10:07:02.164 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.161+0000 7ff4427ee640 1 --2- 172.21.15.62:0/3881112653 >> v2:172.21.15.145:3300/0 conn(0x7ff43c143f00 0x7ff43c144390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:02.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff441fed640 1 --2- 172.21.15.62:0/3881112653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff43c0b9d90 0x7ff43c0b55c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0b5000 tx=0x7ff43801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5432c7aed2dda28d server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff440feb640 1 -- 172.21.15.62:0/3881112653 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4380b0070 con 0x7ff43c0b9d90 2024-04-18T10:07:02.165 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff440feb640 1 -- 172.21.15.62:0/3881112653 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4380a4e50 con 0x7ff43c0b9d90 2024-04-18T10:07:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff440feb640 1 -- 172.21.15.62:0/3881112653 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4380ab820 con 0x7ff43c0b9d90 2024-04-18T10:07:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 -- 172.21.15.62:0/3881112653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff43c0b9d90 msgr2=0x7ff43c0b55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 --2- 172.21.15.62:0/3881112653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff43c0b9d90 0x7ff43c0b55c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0b5000 tx=0x7ff43801c400 comp rx=0 tx=0).stop 2024-04-18T10:07:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 -- 172.21.15.62:0/3881112653 shutdown_connections 2024-04-18T10:07:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 --2- 172.21.15.62:0/3881112653 >> v2:172.21.15.145:3300/0 conn(0x7ff43c143f00 0x7ff43c144390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 --2- 172.21.15.62:0/3881112653 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff43c1480e0 0x7ff43c148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.166 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 --2- 172.21.15.62:0/3881112653 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff43c0b9d90 0x7ff43c0b55c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 -- 172.21.15.62:0/3881112653 >> 172.21.15.62:0/3881112653 conn(0x7ff43c06c0d0 msgr2=0x7ff43c0b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 -- 172.21.15.62:0/3881112653 shutdown_connections 2024-04-18T10:07:02.167 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 -- 172.21.15.62:0/3881112653 wait complete. 2024-04-18T10:07:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 Processor -- start 2024-04-18T10:07:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 -- start start 2024-04-18T10:07:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff43c0b9d90 0x7ff43c142a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff43c143f00 0x7ff43c072b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff43c1480e0 0x7ff43c0b84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c0b47b0 con 0x7ff43c0b9d90 2024-04-18T10:07:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7ff43c0b4630 con 0x7ff43c1480e0 2024-04-18T10:07:02.169 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff444251640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7ff43c0b4930 con 0x7ff43c143f00 2024-04-18T10:07:02.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff441fed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff43c0b9d90 0x7ff43c142a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff441fed640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff43c0b9d90 0x7ff43c142a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35764/0 (socket says 172.21.15.62:35764) 2024-04-18T10:07:02.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff441fed640 1 -- 172.21.15.62:0/668308432 learned_addr learned my addr 172.21.15.62:0/668308432 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:02.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff4417ec640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff43c143f00 0x7ff43c072b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff4427ee640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff43c1480e0 0x7ff43c0b84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff441fed640 1 -- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff43c143f00 msgr2=0x7ff43c072b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff441fed640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff43c143f00 0x7ff43c072b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.170 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff441fed640 1 -- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff43c1480e0 msgr2=0x7ff43c0b84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff441fed640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff43c1480e0 0x7ff43c0b84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.165+0000 7ff441fed640 1 -- 172.21.15.62:0/668308432 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff43c071f20 con 0x7ff43c0b9d90 2024-04-18T10:07:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff441fed640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff43c0b9d90 0x7ff43c142a80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff4380ae040 tx=0x7ff43801ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4327fc640 1 -- 172.21.15.62:0/668308432 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4380ba020 con 0x7ff43c0b9d90 2024-04-18T10:07:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4327fc640 1 -- 172.21.15.62:0/668308432 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4380b6420 con 0x7ff43c0b9d90 2024-04-18T10:07:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff444251640 1 -- 172.21.15.62:0/668308432 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff43c072200 con 0x7ff43c0b9d90 2024-04-18T10:07:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4327fc640 1 -- 172.21.15.62:0/668308432 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4380b0040 con 0x7ff43c0b9d90 2024-04-18T10:07:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff444251640 1 -- 172.21.15.62:0/668308432 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff43c070ef0 con 0x7ff43c0b9d90 2024-04-18T10:07:02.171 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff444251640 1 -- 172.21.15.62:0/668308432 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff43c068ca0 con 0x7ff43c0b9d90 2024-04-18T10:07:02.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4327fc640 1 -- 172.21.15.62:0/668308432 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff4380bd020 con 0x7ff43c0b9d90 2024-04-18T10:07:02.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4327fc640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff420068380 0x7ff42006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4327fc640 1 -- 172.21.15.62:0/668308432 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7ff438120d00 con 0x7ff43c0b9d90 2024-04-18T10:07:02.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4327fc640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] conn(0x7ff42006e240 0x7ff420070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4327fc640 1 -- 172.21.15.62:0/668308432 --> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4380c5e50 con 0x7ff42006e240 2024-04-18T10:07:02.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4327fc640 1 -- 172.21.15.62:0/668308432 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff438121190 con 0x7ff43c0b9d90 2024-04-18T10:07:02.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4417ec640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff420068380 0x7ff42006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.173 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4427ee640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] conn(0x7ff42006e240 0x7ff420070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4417ec640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff420068380 0x7ff42006a840 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff43c1422c0 tx=0x7ff42c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.169+0000 7ff4427ee640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] conn(0x7ff42006e240 0x7ff420070660 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.174 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.173+0000 7ff4327fc640 1 -- 172.21.15.62:0/668308432 <== osd.4 v2:172.21.15.145:6816/4248694380 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4380c5e50 con 0x7ff42006e240 2024-04-18T10:07:02.186 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.185+0000 7ff444251640 1 -- 172.21.15.62:0/668308432 --> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff43c1485f0 con 0x7ff42006e240 2024-04-18T10:07:02.187 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.185+0000 7ff4327fc640 1 -- 172.21.15.62:0/668308432 <== osd.4 v2:172.21.15.145:6816/4248694380 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff43c1485f0 con 0x7ff42006e240 2024-04-18T10:07:02.187 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:07:02.187 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T10:07:02.188 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:07:02.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.185+0000 7ff444251640 1 -- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] conn(0x7ff42006e240 msgr2=0x7ff420070660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.185+0000 7ff444251640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:6816/4248694380,v1:172.21.15.145:6817/4248694380] conn(0x7ff42006e240 0x7ff420070660 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.185+0000 7ff444251640 1 -- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff420068380 msgr2=0x7ff42006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.185+0000 7ff444251640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7ff420068380 0x7ff42006a840 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff43c1422c0 tx=0x7ff42c008040 comp rx=0 tx=0).stop 2024-04-18T10:07:02.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.185+0000 7ff444251640 1 -- 172.21.15.62:0/668308432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff43c0b9d90 msgr2=0x7ff43c142a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.185+0000 7ff444251640 1 --2- 172.21.15.62:0/668308432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff43c0b9d90 0x7ff43c142a80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff4380ae040 tx=0x7ff43801ac40 comp rx=0 tx=0).stop 2024-04-18T10:07:02.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.185+0000 7ff4427ee640 1 -- 172.21.15.62:0/668308432 reap_dead start 2024-04-18T10:07:02.189 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.189+0000 7ff444251640 1 -- 172.21.15.62:0/668308432 shutdown_connections 2024-04-18T10:07:02.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.189+0000 7ff444251640 1 -- 172.21.15.62:0/668308432 >> 172.21.15.62:0/668308432 conn(0x7ff43c06c0d0 msgr2=0x7ff43c14ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:02.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.189+0000 7ff444251640 1 -- 172.21.15.62:0/668308432 shutdown_connections 2024-04-18T10:07:02.190 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.189+0000 7ff444251640 1 -- 172.21.15.62:0/668308432 wait complete. 2024-04-18T10:07:02.202 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-18T10:07:02.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee9915640 1 Processor -- start 2024-04-18T10:07:02.322 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee9915640 1 -- start start 2024-04-18T10:07:02.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee9915640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7ee40b6720 0x7f7ee4141820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee9915640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee414bb30 0x7f7ee4141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee9915640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ee40b55b0 con 0x7f7ee40ba9e0 2024-04-18T10:07:02.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee9915640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f7ee40b76b0 con 0x7f7ee40b6720 2024-04-18T10:07:02.325 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee9915640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee40badc0 con 0x7f7ee414bb30 2024-04-18T10:07:02.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee2ffd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f7ee40ba9e0 0x7f7ee414b420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:53994/0 (socket says 172.21.15.62:53994) 2024-04-18T10:07:02.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee2ffd640 1 -- 172.21.15.62:0/3194278780 learned_addr learned my addr 172.21.15.62:0/3194278780 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:07:02.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee37fe640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee414bb30 0x7f7ee4141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3894830909 0 0) 0x7f7ee40b55b0 con 0x7f7ee40ba9e0 2024-04-18T10:07:02.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ecc003530 con 0x7f7ee40ba9e0 2024-04-18T10:07:02.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee27fc640 1 --2- 172.21.15.62:0/3194278780 >> v2:172.21.15.145:3300/0 conn(0x7f7ee40b6720 0x7f7ee4141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2531881957 0 0) 0x7f7ecc003530 con 0x7f7ee40ba9e0 2024-04-18T10:07:02.326 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee414bb30 msgr2=0x7f7ee4141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:02.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee1ffb640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee414bb30 0x7f7ee4141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 >> v2:172.21.15.145:3300/0 conn(0x7f7ee40b6720 msgr2=0x7f7ee4141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:02.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee1ffb640 1 --2- 172.21.15.62:0/3194278780 >> v2:172.21.15.145:3300/0 conn(0x7f7ee40b6720 0x7f7ee4141820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee40b5850 con 0x7f7ee40ba9e0 2024-04-18T10:07:02.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f7ed4001a20 con 0x7f7ee40ba9e0 2024-04-18T10:07:02.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.321+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 >> v1:172.21.15.62:6789/0 conn(0x7f7ee40ba9e0 legacy=0x7f7ee414b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee1ffb640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee414bb30 0x7f7ecc0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee1ffb640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ecc003f40 0x7f7ecc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee1ffb640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ecc004d50 0x7f7ecc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee40b76b0 con 0x7f7ecc004d50 2024-04-18T10:07:02.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee40badc0 con 0x7f7ecc003f40 2024-04-18T10:07:02.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7ecc00bab0 con 0x7f7ee414bb30 2024-04-18T10:07:02.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee37fe640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ecc004d50 0x7f7ecc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee27fc640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ecc003f40 0x7f7ecc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee37fe640 1 -- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee414bb30 msgr2=0x7f7ecc0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:02.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee2ffd640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee414bb30 0x7f7ecc0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee37fe640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee414bb30 0x7f7ecc0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee37fe640 1 -- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ecc003f40 msgr2=0x7f7ecc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:02.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee37fe640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ecc003f40 0x7f7ecc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee37fe640 1 -- 172.21.15.62:0/3194278780 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ecc003530 con 0x7f7ecc004d50 2024-04-18T10:07:02.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee27fc640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ecc003f40 0x7f7ecc004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:02.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee2ffd640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee414bb30 0x7f7ecc0039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:02.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee37fe640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ecc004d50 0x7f7ecc0051d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009ae0 tx=0x7f7edc01ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c5e6c3704b0696 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc0b2070 con 0x7f7ecc004d50 2024-04-18T10:07:02.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7edc0a5630 con 0x7f7ecc004d50 2024-04-18T10:07:02.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee1ffb640 1 -- 172.21.15.62:0/3194278780 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc0096b0 con 0x7f7ecc004d50 2024-04-18T10:07:02.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee9915640 1 -- 172.21.15.62:0/3194278780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ecc004d50 msgr2=0x7f7ecc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee9915640 1 --2- 172.21.15.62:0/3194278780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ecc004d50 0x7f7ecc0051d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7edc009ae0 tx=0x7f7edc01ca40 comp rx=0 tx=0).stop 2024-04-18T10:07:02.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee37fe640 1 -- 172.21.15.62:0/3194278780 reap_dead start 2024-04-18T10:07:02.333 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee9915640 1 -- 172.21.15.62:0/3194278780 shutdown_connections 2024-04-18T10:07:02.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee9915640 1 -- 172.21.15.62:0/3194278780 >> 172.21.15.62:0/3194278780 conn(0x7f7ee4067e40 msgr2=0x7f7ee40ba070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:02.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee9915640 1 -- 172.21.15.62:0/3194278780 shutdown_connections 2024-04-18T10:07:02.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.325+0000 7f7ee9915640 1 -- 172.21.15.62:0/3194278780 wait complete. 2024-04-18T10:07:02.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee9915640 1 Processor -- start 2024-04-18T10:07:02.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee9915640 1 -- start start 2024-04-18T10:07:02.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee9915640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee41e3bb0 0x7f7ee414c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee9915640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ee413ff10 0x7f7ee4140370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee9915640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ee414eb00 0x7f7ee414ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee9915640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee40b5020 con 0x7f7ee414eb00 2024-04-18T10:07:02.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee9915640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee40b4ea0 con 0x7f7ee413ff10 2024-04-18T10:07:02.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee9915640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f7ee4152d80 con 0x7f7ee41e3bb0 2024-04-18T10:07:02.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ee414eb00 0x7f7ee414ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee37fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ee414eb00 0x7f7ee414ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35794/0 (socket says 172.21.15.62:35794) 2024-04-18T10:07:02.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee37fe640 1 -- 172.21.15.62:0/1402247788 learned_addr learned my addr 172.21.15.62:0/1402247788 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:02.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee2ffd640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee41e3bb0 0x7f7ee414c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee27fc640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ee413ff10 0x7f7ee4140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.335 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee37fe640 1 -- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee41e3bb0 msgr2=0x7f7ee414c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee37fe640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee41e3bb0 0x7f7ee414c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee37fe640 1 -- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ee413ff10 msgr2=0x7f7ee4140370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee37fe640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ee413ff10 0x7f7ee4140370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee37fe640 1 -- 172.21.15.62:0/1402247788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee41e3f90 con 0x7f7ee414eb00 2024-04-18T10:07:02.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee2ffd640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee41e3bb0 0x7f7ee414c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:02.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee27fc640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ee413ff10 0x7f7ee4140370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:02.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee37fe640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ee414eb00 0x7f7ee414ef60 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7edc0b0040 tx=0x7f7edc0197e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.336 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ecbfff640 1 -- 172.21.15.62:0/1402247788 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc0b2050 con 0x7f7ee414eb00 2024-04-18T10:07:02.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ecbfff640 1 -- 172.21.15.62:0/1402247788 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7edc0b3040 con 0x7f7ee414eb00 2024-04-18T10:07:02.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee9915640 1 -- 172.21.15.62:0/1402247788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee40a9fe0 con 0x7f7ee414eb00 2024-04-18T10:07:02.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ecbfff640 1 -- 172.21.15.62:0/1402247788 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7edc0096b0 con 0x7f7ee414eb00 2024-04-18T10:07:02.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee9915640 1 -- 172.21.15.62:0/1402247788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee414df60 con 0x7f7ee414eb00 2024-04-18T10:07:02.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee9915640 1 -- 172.21.15.62:0/1402247788 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ee4068b20 con 0x7f7ee414eb00 2024-04-18T10:07:02.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ecbfff640 1 -- 172.21.15.62:0/1402247788 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7edc0c0030 con 0x7f7ee414eb00 2024-04-18T10:07:02.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ecbfff640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ed0068380 0x7f7ed006a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.337 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ecbfff640 1 -- 172.21.15.62:0/1402247788 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f7edc11fe20 con 0x7f7ee414eb00 2024-04-18T10:07:02.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.329+0000 7f7ee2ffd640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ed0068380 0x7f7ed006a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.333+0000 7f7ee2ffd640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ed0068380 0x7f7ed006a840 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4002240 tx=0x7f7ed4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.338 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.337+0000 7f7ecbfff640 1 -- 172.21.15.62:0/1402247788 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7edc016780 con 0x7f7ee414eb00 2024-04-18T10:07:02.494 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.493+0000 7f7ee9915640 1 -- 172.21.15.62:0/1402247788 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f7ee40b52b0 con 0x7f7ed0068380 2024-04-18T10:07:02.495 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.493+0000 7f7ecbfff640 1 -- 172.21.15.62:0/1402247788 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7ee40b52b0 con 0x7f7ed0068380 2024-04-18T10:07:02.495 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 4 to deep-scrub 2024-04-18T10:07:02.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.497+0000 7f7ee9915640 1 -- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ed0068380 msgr2=0x7f7ed006a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.497+0000 7f7ee9915640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ed0068380 0x7f7ed006a840 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4002240 tx=0x7f7ed4009040 comp rx=0 tx=0).stop 2024-04-18T10:07:02.497 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.497+0000 7f7ee9915640 1 -- 172.21.15.62:0/1402247788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ee414eb00 msgr2=0x7f7ee414ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.497+0000 7f7ee9915640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ee414eb00 0x7f7ee414ef60 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7edc0b0040 tx=0x7f7edc0197e0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.497+0000 7f7ee9915640 1 -- 172.21.15.62:0/1402247788 shutdown_connections 2024-04-18T10:07:02.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.497+0000 7f7ee9915640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f7ed0068380 0x7f7ed006a840 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.497+0000 7f7ee9915640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7ee414eb00 0x7f7ee414ef60 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.497+0000 7f7ee9915640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ee413ff10 0x7f7ee4140370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.497+0000 7f7ee9915640 1 --2- 172.21.15.62:0/1402247788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee41e3bb0 0x7f7ee414c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.497+0000 7f7ee9915640 1 -- 172.21.15.62:0/1402247788 >> 172.21.15.62:0/1402247788 conn(0x7f7ee4067e40 msgr2=0x7f7ee40bbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:02.498 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.497+0000 7f7ee9915640 1 -- 172.21.15.62:0/1402247788 shutdown_connections 2024-04-18T10:07:02.499 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.497+0000 7f7ee9915640 1 -- 172.21.15.62:0/1402247788 wait complete. 2024-04-18T10:07:02.512 INFO:tasks.ceph:Scrubbing osd.5 2024-04-18T10:07:02.513 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-18T10:07:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.625+0000 7fcfae5a0640 1 Processor -- start 2024-04-18T10:07:02.628 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.625+0000 7fcfae5a0640 1 -- start start 2024-04-18T10:07:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfae5a0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcfa80b9d90 0x7fcfa80b55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfae5a0640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcfa81480e0 0x7fcfa8148560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfae5a0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfa80b5b00 con 0x7fcfa8143f00 2024-04-18T10:07:02.629 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfae5a0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa8144a80 con 0x7fcfa80b9d90 2024-04-18T10:07:02.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfae5a0640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa80ba170 con 0x7fcfa81480e0 2024-04-18T10:07:02.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfa7fff640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fcfa80b9d90 0x7fcfa80b55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfacb3d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fcfa8143f00 0x7fcfa8144370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54008/0 (socket says 172.21.15.62:54008) 2024-04-18T10:07:02.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfacb3d640 1 -- 172.21.15.62:0/679726833 learned_addr learned my addr 172.21.15.62:0/679726833 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:07:02.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfa7fff640 1 -- 172.21.15.62:0/679726833 >> v2:172.21.15.145:3300/0 conn(0x7fcfa81480e0 msgr2=0x7fcfa8148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:02.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfa7fff640 1 --2- 172.21.15.62:0/679726833 >> v2:172.21.15.145:3300/0 conn(0x7fcfa81480e0 0x7fcfa8148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfa7fff640 1 -- 172.21.15.62:0/679726833 >> v1:172.21.15.62:6789/0 conn(0x7fcfa8143f00 legacy=0x7fcfa8144370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfa7fff640 1 -- 172.21.15.62:0/679726833 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa809dc10 con 0x7fcfa80b9d90 2024-04-18T10:07:02.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfa7fff640 1 --2- 172.21.15.62:0/679726833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa80b9d90 0x7fcfa80b55c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcfa81491f0 tx=0x7fcf9801e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3601ef699a12ebb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfa6ffd640 1 -- 172.21.15.62:0/679726833 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf980b2070 con 0x7fcfa80b9d90 2024-04-18T10:07:02.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfa6ffd640 1 -- 172.21.15.62:0/679726833 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf980ad600 con 0x7fcfa80b9d90 2024-04-18T10:07:02.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfa6ffd640 1 -- 172.21.15.62:0/679726833 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf980ad9d0 con 0x7fcfa80b9d90 2024-04-18T10:07:02.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfae5a0640 1 -- 172.21.15.62:0/679726833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa80b9d90 msgr2=0x7fcfa80b55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.632 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.629+0000 7fcfae5a0640 1 --2- 172.21.15.62:0/679726833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa80b9d90 0x7fcfa80b55c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fcfa81491f0 tx=0x7fcf9801e570 comp rx=0 tx=0).stop 2024-04-18T10:07:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 -- 172.21.15.62:0/679726833 shutdown_connections 2024-04-18T10:07:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 --2- 172.21.15.62:0/679726833 >> v2:172.21.15.145:3300/0 conn(0x7fcfa81480e0 0x7fcfa8148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 --2- 172.21.15.62:0/679726833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa80b9d90 0x7fcfa80b55c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 -- 172.21.15.62:0/679726833 >> 172.21.15.62:0/679726833 conn(0x7fcfa806c0d0 msgr2=0x7fcfa80b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 -- 172.21.15.62:0/679726833 shutdown_connections 2024-04-18T10:07:02.633 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 -- 172.21.15.62:0/679726833 wait complete. 2024-04-18T10:07:02.635 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 Processor -- start 2024-04-18T10:07:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 -- start start 2024-04-18T10:07:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfa80b9d90 0x7fcfa813d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa8143f00 0x7fcfa8141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa81480e0 0x7fcfa8142230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa806dba0 con 0x7fcfa81480e0 2024-04-18T10:07:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa806da20 con 0x7fcfa80b9d90 2024-04-18T10:07:02.636 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa806dd20 con 0x7fcfa8143f00 2024-04-18T10:07:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfacb3d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa81480e0 0x7fcfa8142230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfacb3d640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa81480e0 0x7fcfa8142230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35802/0 (socket says 172.21.15.62:35802) 2024-04-18T10:07:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfacb3d640 1 -- 172.21.15.62:0/209425450 learned_addr learned my addr 172.21.15.62:0/209425450 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfa7fff640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfa80b9d90 0x7fcfa813d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfa77fe640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa8143f00 0x7fcfa8141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfacb3d640 1 -- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa8143f00 msgr2=0x7fcfa8141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfacb3d640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa8143f00 0x7fcfa8141cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfacb3d640 1 -- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfa80b9d90 msgr2=0x7fcfa813d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.637 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfacb3d640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfa80b9d90 0x7fcfa813d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfacb3d640 1 -- 172.21.15.62:0/209425450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa81439c0 con 0x7fcfa81480e0 2024-04-18T10:07:02.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfa77fe640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa8143f00 0x7fcfa8141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:02.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfa7fff640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfa80b9d90 0x7fcfa813d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:02.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfacb3d640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa81480e0 0x7fcfa8142230 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000cbf0 tx=0x7fcfa000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfa4ff9640 1 -- 172.21.15.62:0/209425450 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0017070 con 0x7fcfa81480e0 2024-04-18T10:07:02.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfa4ff9640 1 -- 172.21.15.62:0/209425450 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa000ad40 con 0x7fcfa81480e0 2024-04-18T10:07:02.638 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfa4ff9640 1 -- 172.21.15.62:0/209425450 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0012600 con 0x7fcfa81480e0 2024-04-18T10:07:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 -- 172.21.15.62:0/209425450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa813dbc0 con 0x7fcfa81480e0 2024-04-18T10:07:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 -- 172.21.15.62:0/209425450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa81479a0 con 0x7fcfa81480e0 2024-04-18T10:07:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.633+0000 7fcfae5a0640 1 -- 172.21.15.62:0/209425450 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcfa8068ca0 con 0x7fcfa81480e0 2024-04-18T10:07:02.639 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.637+0000 7fcfa4ff9640 1 -- 172.21.15.62:0/209425450 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcfa001a070 con 0x7fcfa81480e0 2024-04-18T10:07:02.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.637+0000 7fcfa4ff9640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf94068380 0x7fcf9406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.637+0000 7fcfa4ff9640 1 -- 172.21.15.62:0/209425450 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fcfa0086ad0 con 0x7fcfa81480e0 2024-04-18T10:07:02.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.637+0000 7fcfa4ff9640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] conn(0x7fcf9406e240 0x7fcf94070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.640 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.637+0000 7fcfa4ff9640 1 -- 172.21.15.62:0/209425450 --> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcfa0016ec0 con 0x7fcf9406e240 2024-04-18T10:07:02.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.637+0000 7fcfa4ff9640 1 -- 172.21.15.62:0/209425450 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fcfa0086f60 con 0x7fcfa81480e0 2024-04-18T10:07:02.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.637+0000 7fcfa7fff640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf94068380 0x7fcf9406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.637+0000 7fcfa77fe640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] conn(0x7fcf9406e240 0x7fcf94070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.637+0000 7fcfa7fff640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf94068380 0x7fcf9406a840 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcf980af000 tx=0x7fcf98008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.637+0000 7fcfa77fe640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] conn(0x7fcf9406e240 0x7fcf94070660 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.641 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.637+0000 7fcfa4ff9640 1 -- 172.21.15.62:0/209425450 <== osd.5 v2:172.21.15.145:6824/3413756159 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcfa0016ec0 con 0x7fcf9406e240 2024-04-18T10:07:02.652 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfae5a0640 1 -- 172.21.15.62:0/209425450 --> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcfa81436e0 con 0x7fcf9406e240 2024-04-18T10:07:02.653 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfa4ff9640 1 -- 172.21.15.62:0/209425450 <== osd.5 v2:172.21.15.145:6824/3413756159 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fcfa81436e0 con 0x7fcf9406e240 2024-04-18T10:07:02.653 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:07:02.654 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T10:07:02.654 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:07:02.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfae5a0640 1 -- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] conn(0x7fcf9406e240 msgr2=0x7fcf94070660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfae5a0640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:6824/3413756159,v1:172.21.15.145:6825/3413756159] conn(0x7fcf9406e240 0x7fcf94070660 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfae5a0640 1 -- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf94068380 msgr2=0x7fcf9406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfae5a0640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf94068380 0x7fcf9406a840 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fcf980af000 tx=0x7fcf98008040 comp rx=0 tx=0).stop 2024-04-18T10:07:02.655 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfae5a0640 1 -- 172.21.15.62:0/209425450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa81480e0 msgr2=0x7fcfa8142230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfae5a0640 1 --2- 172.21.15.62:0/209425450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa81480e0 0x7fcfa8142230 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000cbf0 tx=0x7fcfa000a430 comp rx=0 tx=0).stop 2024-04-18T10:07:02.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfacb3d640 1 -- 172.21.15.62:0/209425450 reap_dead start 2024-04-18T10:07:02.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfae5a0640 1 -- 172.21.15.62:0/209425450 shutdown_connections 2024-04-18T10:07:02.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfae5a0640 1 -- 172.21.15.62:0/209425450 >> 172.21.15.62:0/209425450 conn(0x7fcfa806c0d0 msgr2=0x7fcfa8144370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:02.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfae5a0640 1 -- 172.21.15.62:0/209425450 shutdown_connections 2024-04-18T10:07:02.656 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.653+0000 7fcfae5a0640 1 -- 172.21.15.62:0/209425450 wait complete. 2024-04-18T10:07:02.670 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-18T10:07:02.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.785+0000 7f198c9fd640 1 Processor -- start 2024-04-18T10:07:02.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.785+0000 7f198c9fd640 1 -- start start 2024-04-18T10:07:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.785+0000 7f198c9fd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198809c4d0 0x7f198809ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.785+0000 7f198c9fd640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198809d430 con 0x7f198809e2d0 2024-04-18T10:07:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.785+0000 7f198c9fd640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f198809dea0 con 0x7f198809b620 2024-04-18T10:07:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.785+0000 7f198c9fd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f198806eb30 con 0x7f198809c4d0 2024-04-18T10:07:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.785+0000 7f1986d9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f198809e2d0 0x7f1988142ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54016/0 (socket says 172.21.15.62:54016) 2024-04-18T10:07:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.785+0000 7f1986d9d640 1 -- 172.21.15.62:0/2480202506 learned_addr learned my addr 172.21.15.62:0/2480202506 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:07:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.785+0000 7f1985d9b640 1 --2- 172.21.15.62:0/2480202506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198809c4d0 0x7f198809ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.789 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3120534440 0 0) 0x7f198809d430 con 0x7f198809e2d0 2024-04-18T10:07:02.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1974003530 con 0x7f198809e2d0 2024-04-18T10:07:02.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2600656060 0 0) 0x7f1974003530 con 0x7f198809e2d0 2024-04-18T10:07:02.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198809c4d0 msgr2=0x7f198809ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:02.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 --2- 172.21.15.62:0/2480202506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198809c4d0 0x7f198809ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 >> v1:172.21.15.145:6789/0 conn(0x7f198809b620 legacy=0x7f19881492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198809d6a0 con 0x7f198809e2d0 2024-04-18T10:07:02.790 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7f1980001a00 con 0x7f198809e2d0 2024-04-18T10:07:02.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 >> v1:172.21.15.62:6789/0 conn(0x7f198809e2d0 legacy=0x7f1988142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 --2- 172.21.15.62:0/2480202506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19740039f0 0x7f1974003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 --2- 172.21.15.62:0/2480202506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1974004810 0x7f1974004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 --2- 172.21.15.62:0/2480202506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198809c4d0 0x7f1974005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f197400bad0 con 0x7f19740039f0 2024-04-18T10:07:02.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f197400bc80 con 0x7f1974004810 2024-04-18T10:07:02.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f197400be80 con 0x7f198809c4d0 2024-04-18T10:07:02.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198659c640 1 --2- 172.21.15.62:0/2480202506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19740039f0 0x7f1974003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f1985d9b640 1 --2- 172.21.15.62:0/2480202506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1974004810 0x7f1974004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f1986d9d640 1 --2- 172.21.15.62:0/2480202506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198809c4d0 0x7f1974005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198659c640 1 -- 172.21.15.62:0/2480202506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198809c4d0 msgr2=0x7f1974005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:02.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198659c640 1 --2- 172.21.15.62:0/2480202506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198809c4d0 0x7f1974005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198659c640 1 -- 172.21.15.62:0/2480202506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1974004810 msgr2=0x7f1974004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:02.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198659c640 1 --2- 172.21.15.62:0/2480202506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1974004810 0x7f1974004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198659c640 1 -- 172.21.15.62:0/2480202506 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1974003530 con 0x7f19740039f0 2024-04-18T10:07:02.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.789+0000 7f198659c640 1 --2- 172.21.15.62:0/2480202506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19740039f0 0x7f1974003e10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1970001780 tx=0x7f197001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e819dcba53f96b server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19700013a0 con 0x7f19740039f0 2024-04-18T10:07:02.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19700052f0 con 0x7f19740039f0 2024-04-18T10:07:02.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198559a640 1 -- 172.21.15.62:0/2480202506 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970003840 con 0x7f19740039f0 2024-04-18T10:07:02.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 -- 172.21.15.62:0/2480202506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19740039f0 msgr2=0x7f1974003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 --2- 172.21.15.62:0/2480202506 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19740039f0 0x7f1974003e10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1970001780 tx=0x7f197001c420 comp rx=0 tx=0).stop 2024-04-18T10:07:02.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f1986d9d640 1 -- 172.21.15.62:0/2480202506 reap_dead start 2024-04-18T10:07:02.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 -- 172.21.15.62:0/2480202506 shutdown_connections 2024-04-18T10:07:02.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 -- 172.21.15.62:0/2480202506 >> 172.21.15.62:0/2480202506 conn(0x7f19880b9420 msgr2=0x7f19880b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:02.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 -- 172.21.15.62:0/2480202506 shutdown_connections 2024-04-18T10:07:02.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 -- 172.21.15.62:0/2480202506 wait complete. 2024-04-18T10:07:02.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 Processor -- start 2024-04-18T10:07:02.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 -- start start 2024-04-18T10:07:02.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740039f0 0x7f19880b88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19881e1b30 0x7f198813f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198813fc90 0x7f1988098130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1988151a10 con 0x7f19881e1b30 2024-04-18T10:07:02.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f198809bea0 con 0x7f19740039f0 2024-04-18T10:07:02.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198c9fd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f1988151b90 con 0x7f198813fc90 2024-04-18T10:07:02.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f1985d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19881e1b30 0x7f198813f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f1985d9b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19881e1b30 0x7f198813f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35818/0 (socket says 172.21.15.62:35818) 2024-04-18T10:07:02.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f1985d9b640 1 -- 172.21.15.62:0/2933079500 learned_addr learned my addr 172.21.15.62:0/2933079500 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:02.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f198659c640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740039f0 0x7f19880b88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f1986d9d640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198813fc90 0x7f1988098130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f1985d9b640 1 -- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198813fc90 msgr2=0x7f1988098130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f1985d9b640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198813fc90 0x7f1988098130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.799 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f1985d9b640 1 -- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740039f0 msgr2=0x7f19880b88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f1985d9b640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740039f0 0x7f19880b88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f1985d9b640 1 -- 172.21.15.62:0/2933079500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1988073990 con 0x7f19881e1b30 2024-04-18T10:07:02.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f1986d9d640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198813fc90 0x7f1988098130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:07:02.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f1985d9b640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19881e1b30 0x7f198813f750 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1978011590 tx=0x7f1978011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f197effd640 1 -- 172.21.15.62:0/2933079500 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1978004220 con 0x7f19881e1b30 2024-04-18T10:07:02.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.793+0000 7f197effd640 1 -- 172.21.15.62:0/2933079500 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19780043c0 con 0x7f19881e1b30 2024-04-18T10:07:02.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.797+0000 7f197effd640 1 -- 172.21.15.62:0/2933079500 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1978015870 con 0x7f19881e1b30 2024-04-18T10:07:02.800 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.797+0000 7f198c9fd640 1 -- 172.21.15.62:0/2933079500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19880b7bd0 con 0x7f19881e1b30 2024-04-18T10:07:02.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.797+0000 7f198c9fd640 1 -- 172.21.15.62:0/2933079500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19880b7e40 con 0x7f19881e1b30 2024-04-18T10:07:02.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.797+0000 7f198c9fd640 1 -- 172.21.15.62:0/2933079500 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1988068b20 con 0x7f19881e1b30 2024-04-18T10:07:02.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.797+0000 7f197effd640 1 -- 172.21.15.62:0/2933079500 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1978015a10 con 0x7f19881e1b30 2024-04-18T10:07:02.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.797+0000 7f197effd640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f19680683b0 0x7f196806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:02.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.797+0000 7f197effd640 1 -- 172.21.15.62:0/2933079500 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f1978086710 con 0x7f19881e1b30 2024-04-18T10:07:02.801 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.797+0000 7f198659c640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f19680683b0 0x7f196806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:02.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.797+0000 7f198659c640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f19680683b0 0x7f196806a870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1970001750 tx=0x7f19700011a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:02.802 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.801+0000 7f197effd640 1 -- 172.21.15.62:0/2933079500 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1978019030 con 0x7f19881e1b30 2024-04-18T10:07:02.961 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.960+0000 7f198c9fd640 1 -- 172.21.15.62:0/2933079500 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f198813f450 con 0x7f19680683b0 2024-04-18T10:07:02.962 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.960+0000 7f197effd640 1 -- 172.21.15.62:0/2933079500 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1970005150 con 0x7f19680683b0 2024-04-18T10:07:02.962 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 5 to deep-scrub 2024-04-18T10:07:02.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.964+0000 7f198c9fd640 1 -- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f19680683b0 msgr2=0x7f196806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.964+0000 7f198c9fd640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f19680683b0 0x7f196806a870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1970001750 tx=0x7f19700011a0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.964+0000 7f198c9fd640 1 -- 172.21.15.62:0/2933079500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19881e1b30 msgr2=0x7f198813f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:02.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.964+0000 7f198c9fd640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19881e1b30 0x7f198813f750 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1978011590 tx=0x7f1978011a50 comp rx=0 tx=0).stop 2024-04-18T10:07:02.964 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.964+0000 7f198c9fd640 1 -- 172.21.15.62:0/2933079500 shutdown_connections 2024-04-18T10:07:02.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.964+0000 7f198c9fd640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f19680683b0 0x7f196806a870 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.964+0000 7f198c9fd640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f198813fc90 0x7f1988098130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.964+0000 7f198c9fd640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19881e1b30 0x7f198813f750 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.964+0000 7f198c9fd640 1 --2- 172.21.15.62:0/2933079500 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740039f0 0x7f19880b88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:02.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.964+0000 7f198c9fd640 1 -- 172.21.15.62:0/2933079500 >> 172.21.15.62:0/2933079500 conn(0x7f19880b9420 msgr2=0x7f1988143cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:02.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.964+0000 7f198c9fd640 1 -- 172.21.15.62:0/2933079500 shutdown_connections 2024-04-18T10:07:02.965 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:02.964+0000 7f198c9fd640 1 -- 172.21.15.62:0/2933079500 wait complete. 2024-04-18T10:07:02.980 INFO:tasks.ceph:Scrubbing osd.6 2024-04-18T10:07:02.980 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-18T10:07:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d4ddaa640 1 Processor -- start 2024-04-18T10:07:03.117 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d4ddaa640 1 -- start start 2024-04-18T10:07:03.118 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d4ddaa640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2d480bbef0 0x7f2d48141e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d4ddaa640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2d4806cbe0 0x7f2d4806e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d4ddaa640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d480b7e50 con 0x7f2d4806e8c0 2024-04-18T10:07:03.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d4ddaa640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f2d4806dc10 con 0x7f2d4806cbe0 2024-04-18T10:07:03.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d4ddaa640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f2d480b7bb0 con 0x7f2d480bbef0 2024-04-18T10:07:03.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d46ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2d4806cbe0 0x7f2d4806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d46ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f2d4806cbe0 0x7f2d4806e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35826/0 (socket says 172.21.15.62:35826) 2024-04-18T10:07:03.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d46ffd640 1 -- 172.21.15.62:0/857020887 learned_addr learned my addr 172.21.15.62:0/857020887 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:03.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d477fe640 1 --2- 172.21.15.62:0/857020887 >> v2:172.21.15.145:3300/0 conn(0x7f2d480bbef0 0x7f2d48141e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.119 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d46ffd640 1 -- 172.21.15.62:0/857020887 >> v2:172.21.15.145:3300/0 conn(0x7f2d480bbef0 msgr2=0x7f2d48141e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:03.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d46ffd640 1 --2- 172.21.15.62:0/857020887 >> v2:172.21.15.145:3300/0 conn(0x7f2d480bbef0 0x7f2d48141e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d46ffd640 1 -- 172.21.15.62:0/857020887 >> v1:172.21.15.145:6789/0 conn(0x7f2d4806e8c0 legacy=0x7f2d480725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:03.120 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.116+0000 7f2d46ffd640 1 -- 172.21.15.62:0/857020887 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d480b53a0 con 0x7f2d4806cbe0 2024-04-18T10:07:03.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d46ffd640 1 --2- 172.21.15.62:0/857020887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4806cbe0 0x7f2d4806e380 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2d48073d50 tx=0x7f2d3801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe2ffc1c07f8c9f server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d467fc640 1 -- 172.21.15.62:0/857020887 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d380b0070 con 0x7f2d4806cbe0 2024-04-18T10:07:03.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d467fc640 1 -- 172.21.15.62:0/857020887 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d380ab600 con 0x7f2d4806cbe0 2024-04-18T10:07:03.121 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d467fc640 1 -- 172.21.15.62:0/857020887 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d380ab9d0 con 0x7f2d4806cbe0 2024-04-18T10:07:03.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/857020887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4806cbe0 msgr2=0x7f2d4806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 --2- 172.21.15.62:0/857020887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4806cbe0 0x7f2d4806e380 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2d48073d50 tx=0x7f2d3801c490 comp rx=0 tx=0).stop 2024-04-18T10:07:03.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/857020887 shutdown_connections 2024-04-18T10:07:03.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 --2- 172.21.15.62:0/857020887 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d4806cbe0 0x7f2d4806e380 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 --2- 172.21.15.62:0/857020887 >> v2:172.21.15.145:3300/0 conn(0x7f2d480bbef0 0x7f2d48141e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/857020887 >> 172.21.15.62:0/857020887 conn(0x7f2d48067fd0 msgr2=0x7f2d480bb540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:03.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/857020887 shutdown_connections 2024-04-18T10:07:03.123 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/857020887 wait complete. 2024-04-18T10:07:03.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 Processor -- start 2024-04-18T10:07:03.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 -- start start 2024-04-18T10:07:03.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d4806cbe0 0x7f2d48138610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d4806e8c0 0x7f2d481e0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d480bbef0 0x7f2d481e0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f2d48142500 con 0x7f2d480bbef0 2024-04-18T10:07:03.125 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f2d48142380 con 0x7f2d4806cbe0 2024-04-18T10:07:03.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d4ddaa640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f2d48155630 con 0x7f2d4806e8c0 2024-04-18T10:07:03.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d47fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d480bbef0 0x7f2d481e0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d47fff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d480bbef0 0x7f2d481e0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35838/0 (socket says 172.21.15.62:35838) 2024-04-18T10:07:03.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d47fff640 1 -- 172.21.15.62:0/4162462229 learned_addr learned my addr 172.21.15.62:0/4162462229 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:03.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.120+0000 7f2d46ffd640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d4806e8c0 0x7f2d481e0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d477fe640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d4806cbe0 0x7f2d48138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d47fff640 1 -- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d4806e8c0 msgr2=0x7f2d481e0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d47fff640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d4806e8c0 0x7f2d481e0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.126 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d47fff640 1 -- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d4806cbe0 msgr2=0x7f2d48138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d47fff640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d4806cbe0 0x7f2d48138610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d47fff640 1 -- 172.21.15.62:0/4162462229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d480694c0 con 0x7f2d480bbef0 2024-04-18T10:07:03.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d47fff640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d480bbef0 0x7f2d481e0e50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2d4000c770 tx=0x7f2d4000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d3ffff640 1 -- 172.21.15.62:0/4162462229 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d40003b80 con 0x7f2d480bbef0 2024-04-18T10:07:03.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d3ffff640 1 -- 172.21.15.62:0/4162462229 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d40003d20 con 0x7f2d480bbef0 2024-04-18T10:07:03.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d3ffff640 1 -- 172.21.15.62:0/4162462229 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d4000aa90 con 0x7f2d480bbef0 2024-04-18T10:07:03.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/4162462229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d48145af0 con 0x7f2d480bbef0 2024-04-18T10:07:03.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/4162462229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d48139c50 con 0x7f2d480bbef0 2024-04-18T10:07:03.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/4162462229 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2d48068ca0 con 0x7f2d480bbef0 2024-04-18T10:07:03.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d3ffff640 1 -- 172.21.15.62:0/4162462229 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2d4000ac30 con 0x7f2d480bbef0 2024-04-18T10:07:03.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d3ffff640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2d28068380 0x7f2d2806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d3ffff640 1 -- 172.21.15.62:0/4162462229 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f2d40086fd0 con 0x7f2d480bbef0 2024-04-18T10:07:03.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d3ffff640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] conn(0x7f2d2806e240 0x7f2d28070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d3ffff640 1 -- 172.21.15.62:0/4162462229 --> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d40023e80 con 0x7f2d2806e240 2024-04-18T10:07:03.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d3ffff640 1 -- 172.21.15.62:0/4162462229 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f2d40087460 con 0x7f2d480bbef0 2024-04-18T10:07:03.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d477fe640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2d28068380 0x7f2d2806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d46ffd640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] conn(0x7f2d2806e240 0x7f2d28070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d477fe640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2d28068380 0x7f2d2806a840 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2d3000b930 tx=0x7f2d30008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.124+0000 7f2d46ffd640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] conn(0x7f2d2806e240 0x7f2d28070660 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.128+0000 7f2d3ffff640 1 -- 172.21.15.62:0/4162462229 <== osd.6 v2:172.21.15.145:6800/2070684678 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2d40023e80 con 0x7f2d2806e240 2024-04-18T10:07:03.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.140+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/4162462229 --> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2d4806d650 con 0x7f2d2806e240 2024-04-18T10:07:03.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.140+0000 7f2d3ffff640 1 -- 172.21.15.62:0/4162462229 <== osd.6 v2:172.21.15.145:6800/2070684678 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2d4806d650 con 0x7f2d2806e240 2024-04-18T10:07:03.143 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:07:03.144 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T10:07:03.144 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:07:03.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.144+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] conn(0x7f2d2806e240 msgr2=0x7f2d28070660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.144+0000 7f2d4ddaa640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:6800/2070684678,v1:172.21.15.145:6801/2070684678] conn(0x7f2d2806e240 0x7f2d28070660 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.144+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2d28068380 msgr2=0x7f2d2806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.144+0000 7f2d4ddaa640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f2d28068380 0x7f2d2806a840 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2d3000b930 tx=0x7f2d30008040 comp rx=0 tx=0).stop 2024-04-18T10:07:03.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.144+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/4162462229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d480bbef0 msgr2=0x7f2d481e0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.144+0000 7f2d4ddaa640 1 --2- 172.21.15.62:0/4162462229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2d480bbef0 0x7f2d481e0e50 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f2d4000c770 tx=0x7f2d4000cc30 comp rx=0 tx=0).stop 2024-04-18T10:07:03.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.144+0000 7f2d47fff640 1 -- 172.21.15.62:0/4162462229 reap_dead start 2024-04-18T10:07:03.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.144+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/4162462229 shutdown_connections 2024-04-18T10:07:03.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.144+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/4162462229 >> 172.21.15.62:0/4162462229 conn(0x7f2d48067fd0 msgr2=0x7f2d48074a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:03.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.144+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/4162462229 shutdown_connections 2024-04-18T10:07:03.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.144+0000 7f2d4ddaa640 1 -- 172.21.15.62:0/4162462229 wait complete. 2024-04-18T10:07:03.159 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-18T10:07:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8ce1a0640 1 Processor -- start 2024-04-18T10:07:03.272 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8ce1a0640 1 -- start start 2024-04-18T10:07:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8ce1a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8c80ba9e0 0x7fb8c814b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.273 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8ce1a0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fb8c80b6720 0x7fb8c814b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8ce1a0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8c80b55b0 con 0x7fb8c8141820 2024-04-18T10:07:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8ce1a0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fb8c80b76b0 con 0x7fb8c80b6720 2024-04-18T10:07:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8ce1a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fb8c80badc0 con 0x7fb8c80ba9e0 2024-04-18T10:07:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c7fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb8c8141820 0x7fb8c8141c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54032/0 (socket says 172.21.15.62:54032) 2024-04-18T10:07:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c7fff640 1 -- 172.21.15.62:0/3744684154 learned_addr learned my addr 172.21.15.62:0/3744684154 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:07:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c6ffd640 1 --2- 172.21.15.62:0/3744684154 >> v2:172.21.15.62:3300/0 conn(0x7fb8c80b6720 0x7fb8c814b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c77fe640 1 --2- 172.21.15.62:0/3744684154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8c80ba9e0 0x7fb8c814b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.274 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c67fc640 1 -- 172.21.15.62:0/3744684154 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 821438429 0 0) 0x7fb8c80b55b0 con 0x7fb8c8141820 2024-04-18T10:07:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c67fc640 1 -- 172.21.15.62:0/3744684154 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8b4003530 con 0x7fb8c8141820 2024-04-18T10:07:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c6ffd640 1 -- 172.21.15.62:0/3744684154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8c80ba9e0 msgr2=0x7fb8c814b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c6ffd640 1 --2- 172.21.15.62:0/3744684154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8c80ba9e0 0x7fb8c814b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c6ffd640 1 -- 172.21.15.62:0/3744684154 >> v1:172.21.15.62:6789/0 conn(0x7fb8c8141820 legacy=0x7fb8c8141c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.275 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c6ffd640 1 -- 172.21.15.62:0/3744684154 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c80b5850 con 0x7fb8c80b6720 2024-04-18T10:07:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c6ffd640 1 --2- 172.21.15.62:0/3744684154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8c80b6720 0x7fb8c814b960 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00097c0 tx=0x7fb8b001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7589b2c97080dbe server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c67fc640 1 -- 172.21.15.62:0/3744684154 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b00b0070 con 0x7fb8c80b6720 2024-04-18T10:07:03.276 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c67fc640 1 -- 172.21.15.62:0/3744684154 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8b00ab600 con 0x7fb8c80b6720 2024-04-18T10:07:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.272+0000 7fb8c67fc640 1 -- 172.21.15.62:0/3744684154 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b00ab9b0 con 0x7fb8c80b6720 2024-04-18T10:07:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3744684154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8c80b6720 msgr2=0x7fb8c814b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 --2- 172.21.15.62:0/3744684154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8c80b6720 0x7fb8c814b960 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00097c0 tx=0x7fb8b001c380 comp rx=0 tx=0).stop 2024-04-18T10:07:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3744684154 shutdown_connections 2024-04-18T10:07:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 --2- 172.21.15.62:0/3744684154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8c80b6720 0x7fb8c814b960 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 --2- 172.21.15.62:0/3744684154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8c80ba9e0 0x7fb8c814b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3744684154 >> 172.21.15.62:0/3744684154 conn(0x7fb8c8067e40 msgr2=0x7fb8c80ba070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3744684154 shutdown_connections 2024-04-18T10:07:03.277 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3744684154 wait complete. 2024-04-18T10:07:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 Processor -- start 2024-04-18T10:07:03.279 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 -- start start 2024-04-18T10:07:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8c80b6720 0x7fb8c814c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8c80ba9e0 0x7fb8c814dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8c8141820 0x7fb8c81e0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c8154d90 con 0x7fb8c80b6720 2024-04-18T10:07:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c8154f10 con 0x7fb8c8141820 2024-04-18T10:07:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fb8c8155090 con 0x7fb8c80ba9e0 2024-04-18T10:07:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8c77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8c80b6720 0x7fb8c814c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.280 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8c77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8c80b6720 0x7fb8c814c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35864/0 (socket says 172.21.15.62:35864) 2024-04-18T10:07:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8c77fe640 1 -- 172.21.15.62:0/3081616398 learned_addr learned my addr 172.21.15.62:0/3081616398 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8c6ffd640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8c80ba9e0 0x7fb8c814dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8c77fe640 1 -- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8c80ba9e0 msgr2=0x7fb8c814dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8c77fe640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8c80ba9e0 0x7fb8c814dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8c77fe640 1 -- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8c8141820 msgr2=0x7fb8c81e0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8c77fe640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8c8141820 0x7fb8c81e0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8c77fe640 1 -- 172.21.15.62:0/3081616398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c80be270 con 0x7fb8c80b6720 2024-04-18T10:07:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8c77fe640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8c80b6720 0x7fb8c814c610 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800e930 tx=0x7fb8b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.281 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8bffff640 1 -- 172.21.15.62:0/3081616398 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8017070 con 0x7fb8c80b6720 2024-04-18T10:07:03.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8bffff640 1 -- 172.21.15.62:0/3081616398 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8b800ca00 con 0x7fb8c80b6720 2024-04-18T10:07:03.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3081616398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8c814f1c0 con 0x7fb8c80b6720 2024-04-18T10:07:03.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8bffff640 1 -- 172.21.15.62:0/3081616398 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b800cdb0 con 0x7fb8c80b6720 2024-04-18T10:07:03.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3081616398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8c814e300 con 0x7fb8c80b6720 2024-04-18T10:07:03.282 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.276+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3081616398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8c8068b20 con 0x7fb8c80b6720 2024-04-18T10:07:03.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.280+0000 7fb8bffff640 1 -- 172.21.15.62:0/3081616398 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb8b801a070 con 0x7fb8c80b6720 2024-04-18T10:07:03.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.280+0000 7fb8bffff640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb8a8068380 0x7fb8a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.280+0000 7fb8bffff640 1 -- 172.21.15.62:0/3081616398 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fb8b8086e00 con 0x7fb8c80b6720 2024-04-18T10:07:03.283 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.280+0000 7fb8c6ffd640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb8a8068380 0x7fb8a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.280+0000 7fb8c6ffd640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb8a8068380 0x7fb8a806a840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00ad000 tx=0x7fb8b0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.284 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.284+0000 7fb8bffff640 1 -- 172.21.15.62:0/3081616398 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb8b80578f0 con 0x7fb8c80b6720 2024-04-18T10:07:03.441 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.440+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3081616398 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fb8c8147050 con 0x7fb8a8068380 2024-04-18T10:07:03.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.440+0000 7fb8bffff640 1 -- 172.21.15.62:0/3081616398 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb8c8147050 con 0x7fb8a8068380 2024-04-18T10:07:03.442 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 6 to deep-scrub 2024-04-18T10:07:03.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.444+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb8a8068380 msgr2=0x7fb8a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.444+0000 7fb8ce1a0640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb8a8068380 0x7fb8a806a840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb8b00ad000 tx=0x7fb8b0008040 comp rx=0 tx=0).stop 2024-04-18T10:07:03.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.444+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3081616398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8c80b6720 msgr2=0x7fb8c814c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.444+0000 7fb8ce1a0640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8c80b6720 0x7fb8c814c610 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb8b800e930 tx=0x7fb8b800edf0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.444 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.444+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3081616398 shutdown_connections 2024-04-18T10:07:03.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.444+0000 7fb8ce1a0640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fb8a8068380 0x7fb8a806a840 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.444+0000 7fb8ce1a0640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb8c8141820 0x7fb8c81e0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.444+0000 7fb8ce1a0640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8c80ba9e0 0x7fb8c814dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.444+0000 7fb8ce1a0640 1 --2- 172.21.15.62:0/3081616398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb8c80b6720 0x7fb8c814c610 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.444+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3081616398 >> 172.21.15.62:0/3081616398 conn(0x7fb8c8067e40 msgr2=0x7fb8c8147a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:03.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.444+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3081616398 shutdown_connections 2024-04-18T10:07:03.445 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.444+0000 7fb8ce1a0640 1 -- 172.21.15.62:0/3081616398 wait complete. 2024-04-18T10:07:03.459 INFO:tasks.ceph:Scrubbing osd.7 2024-04-18T10:07:03.459 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-18T10:07:03.574 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.572+0000 7f1fa065e640 1 Processor -- start 2024-04-18T10:07:03.575 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.572+0000 7f1fa065e640 1 -- start start 2024-04-18T10:07:03.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.572+0000 7f1fa065e640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1f981493f0 0x7f1f98149880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.572+0000 7f1fa065e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f98149dc0 0x7f1f981508c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.572+0000 7f1fa065e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f9814a210 con 0x7f1f980bbf20 2024-04-18T10:07:03.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.572+0000 7f1fa065e640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f1f980b7eb0 con 0x7f1f981493f0 2024-04-18T10:07:03.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.572+0000 7f1fa065e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f1f980b3ae0 con 0x7f1f98149dc0 2024-04-18T10:07:03.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.572+0000 7f1f9dbf9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1f981493f0 0x7f1f98149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.576 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.572+0000 7f1f9dbf9640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f1f981493f0 0x7f1f98149880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35878/0 (socket says 172.21.15.62:35878) 2024-04-18T10:07:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.572+0000 7f1f9dbf9640 1 -- 172.21.15.62:0/2627851450 learned_addr learned my addr 172.21.15.62:0/2627851450 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1f9ebfb640 1 --2- 172.21.15.62:0/2627851450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f98149dc0 0x7f1f981508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1f9dbf9640 1 -- 172.21.15.62:0/2627851450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f98149dc0 msgr2=0x7f1f981508c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1f9dbf9640 1 --2- 172.21.15.62:0/2627851450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f98149dc0 0x7f1f981508c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1f9dbf9640 1 -- 172.21.15.62:0/2627851450 >> v1:172.21.15.145:6789/0 conn(0x7f1f980bbf20 legacy=0x7f1f980b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1f9dbf9640 1 -- 172.21.15.62:0/2627851450 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f980bc300 con 0x7f1f981493f0 2024-04-18T10:07:03.577 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1f9ebfb640 1 --2- 172.21.15.62:0/2627851450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f98149dc0 0x7f1f981508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:07:03.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1f9dbf9640 1 --2- 172.21.15.62:0/2627851450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f981493f0 0x7f1f98149880 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1f88009870 tx=0x7f1f8801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5434bfeb5576598c server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1f9d3f8640 1 -- 172.21.15.62:0/2627851450 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f880b0070 con 0x7f1f981493f0 2024-04-18T10:07:03.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1f9d3f8640 1 -- 172.21.15.62:0/2627851450 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f880ab600 con 0x7f1f981493f0 2024-04-18T10:07:03.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1f9d3f8640 1 -- 172.21.15.62:0/2627851450 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f880ab9d0 con 0x7f1f981493f0 2024-04-18T10:07:03.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1fa065e640 1 -- 172.21.15.62:0/2627851450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f981493f0 msgr2=0x7f1f98149880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1fa065e640 1 --2- 172.21.15.62:0/2627851450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f981493f0 0x7f1f98149880 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1f88009870 tx=0x7f1f8801c440 comp rx=0 tx=0).stop 2024-04-18T10:07:03.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1fa065e640 1 -- 172.21.15.62:0/2627851450 shutdown_connections 2024-04-18T10:07:03.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1fa065e640 1 --2- 172.21.15.62:0/2627851450 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f98149dc0 0x7f1f981508c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1fa065e640 1 --2- 172.21.15.62:0/2627851450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f981493f0 0x7f1f98149880 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1fa065e640 1 -- 172.21.15.62:0/2627851450 >> 172.21.15.62:0/2627851450 conn(0x7f1f98067de0 msgr2=0x7f1f980bb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:03.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1fa065e640 1 -- 172.21.15.62:0/2627851450 shutdown_connections 2024-04-18T10:07:03.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.576+0000 7f1fa065e640 1 -- 172.21.15.62:0/2627851450 wait complete. 2024-04-18T10:07:03.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1fa065e640 1 Processor -- start 2024-04-18T10:07:03.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1fa065e640 1 -- start start 2024-04-18T10:07:03.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1fa065e640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f980bbf20 0x7f1f981443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1fa065e640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f981493f0 0x7f1f9806e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1fa065e640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f98149dc0 0x7f1f9806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1fa065e640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f1f980b5090 con 0x7f1f98149dc0 2024-04-18T10:07:03.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1fa065e640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f1f980b6380 con 0x7f1f981493f0 2024-04-18T10:07:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1fa065e640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f1f980b5210 con 0x7f1f980bbf20 2024-04-18T10:07:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f9ebfb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f98149dc0 0x7f1f9806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f9ebfb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f98149dc0 0x7f1f9806cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35886/0 (socket says 172.21.15.62:35886) 2024-04-18T10:07:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f9ebfb640 1 -- 172.21.15.62:0/3041781956 learned_addr learned my addr 172.21.15.62:0/3041781956 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f9e3fa640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f980bbf20 0x7f1f981443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f9dbf9640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f981493f0 0x7f1f9806e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f9ebfb640 1 -- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f980bbf20 msgr2=0x7f1f981443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f9ebfb640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f980bbf20 0x7f1f981443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f9ebfb640 1 -- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f981493f0 msgr2=0x7f1f9806e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f9ebfb640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f981493f0 0x7f1f9806e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f9ebfb640 1 -- 172.21.15.62:0/3041781956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f9806ebc0 con 0x7f1f98149dc0 2024-04-18T10:07:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f9dbf9640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f981493f0 0x7f1f9806e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f9ebfb640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f98149dc0 0x7f1f9806cbe0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1f90007e70 tx=0x7f1f9000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f8effd640 1 -- 172.21.15.62:0/3041781956 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f9001a070 con 0x7f1f98149dc0 2024-04-18T10:07:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f8effd640 1 -- 172.21.15.62:0/3041781956 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f90003bc0 con 0x7f1f98149dc0 2024-04-18T10:07:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1fa065e640 1 -- 172.21.15.62:0/3041781956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f980b1760 con 0x7f1f98149dc0 2024-04-18T10:07:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f8effd640 1 -- 172.21.15.62:0/3041781956 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f90004030 con 0x7f1f98149dc0 2024-04-18T10:07:03.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1fa065e640 1 -- 172.21.15.62:0/3041781956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f980b2240 con 0x7f1f98149dc0 2024-04-18T10:07:03.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1fa065e640 1 -- 172.21.15.62:0/3041781956 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1f98068ca0 con 0x7f1f98149dc0 2024-04-18T10:07:03.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f8effd640 1 -- 172.21.15.62:0/3041781956 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1f9001d070 con 0x7f1f98149dc0 2024-04-18T10:07:03.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f8effd640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1f7c068380 0x7f1f7c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.580+0000 7f1f8effd640 1 -- 172.21.15.62:0/3041781956 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f1f90086f40 con 0x7f1f98149dc0 2024-04-18T10:07:03.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.584+0000 7f1f8effd640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] conn(0x7f1f7c06e240 0x7f1f7c070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.584+0000 7f1f8effd640 1 -- 172.21.15.62:0/3041781956 --> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1f90012e50 con 0x7f1f7c06e240 2024-04-18T10:07:03.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.584+0000 7f1f8effd640 1 -- 172.21.15.62:0/3041781956 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f1f900873d0 con 0x7f1f98149dc0 2024-04-18T10:07:03.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.584+0000 7f1f9e3fa640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1f7c068380 0x7f1f7c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.584+0000 7f1f9dbf9640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] conn(0x7f1f7c06e240 0x7f1f7c070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.584+0000 7f1f9dbf9640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] conn(0x7f1f7c06e240 0x7f1f7c070660 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.584+0000 7f1f9e3fa640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1f7c068380 0x7f1f7c06a840 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1f94000f70 tx=0x7f1f94009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.588 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.584+0000 7f1f8effd640 1 -- 172.21.15.62:0/3041781956 <== osd.7 v2:172.21.15.145:6802/69473053 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1f90012e50 con 0x7f1f7c06e240 2024-04-18T10:07:03.599 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.596+0000 7f1fa065e640 1 -- 172.21.15.62:0/3041781956 --> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1f9806e3a0 con 0x7f1f7c06e240 2024-04-18T10:07:03.600 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.600+0000 7f1f8effd640 1 -- 172.21.15.62:0/3041781956 <== osd.7 v2:172.21.15.145:6802/69473053 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1f88009490 con 0x7f1f7c06e240 2024-04-18T10:07:03.600 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-04-18T10:07:03.600 INFO:teuthology.orchestra.run.smithi062.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-18T10:07:03.600 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-04-18T10:07:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.600+0000 7f1fa065e640 1 -- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] conn(0x7f1f7c06e240 msgr2=0x7f1f7c070660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.600+0000 7f1fa065e640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:6802/69473053,v1:172.21.15.145:6804/69473053] conn(0x7f1f7c06e240 0x7f1f7c070660 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.600+0000 7f1fa065e640 1 -- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1f7c068380 msgr2=0x7f1f7c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.600+0000 7f1fa065e640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f1f7c068380 0x7f1f7c06a840 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1f94000f70 tx=0x7f1f94009040 comp rx=0 tx=0).stop 2024-04-18T10:07:03.601 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.600+0000 7f1fa065e640 1 -- 172.21.15.62:0/3041781956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f98149dc0 msgr2=0x7f1f9806cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.600+0000 7f1fa065e640 1 --2- 172.21.15.62:0/3041781956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1f98149dc0 0x7f1f9806cbe0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1f90007e70 tx=0x7f1f9000d4a0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.600+0000 7f1f9ebfb640 1 -- 172.21.15.62:0/3041781956 reap_dead start 2024-04-18T10:07:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.600+0000 7f1fa065e640 1 -- 172.21.15.62:0/3041781956 shutdown_connections 2024-04-18T10:07:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.600+0000 7f1fa065e640 1 -- 172.21.15.62:0/3041781956 >> 172.21.15.62:0/3041781956 conn(0x7f1f98067de0 msgr2=0x7f1f980bae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.600+0000 7f1fa065e640 1 -- 172.21.15.62:0/3041781956 shutdown_connections 2024-04-18T10:07:03.602 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.600+0000 7f1fa065e640 1 -- 172.21.15.62:0/3041781956 wait complete. 2024-04-18T10:07:03.615 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-18T10:07:03.727 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.724+0000 7fcfa7c30640 1 Processor -- start 2024-04-18T10:07:03.728 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.724+0000 7fcfa7c30640 1 -- start start 2024-04-18T10:07:03.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa7c30640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa009b620 0x7fcfa009ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa7c30640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfa009d430 con 0x7fcfa009e2d0 2024-04-18T10:07:03.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa7c30640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfa009dea0 con 0x7fcfa009c4d0 2024-04-18T10:07:03.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa7c30640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa006eb30 con 0x7fcfa009b620 2024-04-18T10:07:03.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa61cd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fcfa009e2d0 0x7fcfa0142ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54038/0 (socket says 172.21.15.62:54038) 2024-04-18T10:07:03.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa61cd640 1 -- 172.21.15.62:0/3830457968 learned_addr learned my addr 172.21.15.62:0/3830457968 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:07:03.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa59cc640 1 --2- 172.21.15.62:0/3830457968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa009b620 0x7fcfa009ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464046183 0 0) 0x7fcfa009d430 con 0x7fcfa009e2d0 2024-04-18T10:07:03.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf8c003530 con 0x7fcfa009e2d0 2024-04-18T10:07:03.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 <== mon.0 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2316498937 0 0) 0x7fcf8c003530 con 0x7fcfa009e2d0 2024-04-18T10:07:03.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa009b620 msgr2=0x7fcfa009ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:03.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 --2- 172.21.15.62:0/3830457968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa009b620 0x7fcfa009ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 >> v1:172.21.15.145:6789/0 conn(0x7fcfa009c4d0 legacy=0x7fcfa01492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa009d6a0 con 0x7fcfa009e2d0 2024-04-18T10:07:03.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 <== mon.0 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 726098357 0 0) 0x7fcf98001a00 con 0x7fcfa009e2d0 2024-04-18T10:07:03.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 >> v1:172.21.15.62:6789/0 conn(0x7fcfa009e2d0 legacy=0x7fcfa0142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 --2- 172.21.15.62:0/3830457968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf8c0039f0 0x7fcf8c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 --2- 172.21.15.62:0/3830457968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa009b620 0x7fcf8c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 --2- 172.21.15.62:0/3830457968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf8c004d50 0x7fcf8c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c00bad0 con 0x7fcf8c004d50 2024-04-18T10:07:03.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c00bc80 con 0x7fcf8c0039f0 2024-04-18T10:07:03.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fcf8c00be80 con 0x7fcfa009b620 2024-04-18T10:07:03.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa61cd640 1 --2- 172.21.15.62:0/3830457968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf8c004d50 0x7fcf8c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa59cc640 1 --2- 172.21.15.62:0/3830457968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf8c0039f0 0x7fcf8c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa51cb640 1 --2- 172.21.15.62:0/3830457968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa009b620 0x7fcf8c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa61cd640 1 -- 172.21.15.62:0/3830457968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa009b620 msgr2=0x7fcf8c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:03.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa61cd640 1 --2- 172.21.15.62:0/3830457968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa009b620 0x7fcf8c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa61cd640 1 -- 172.21.15.62:0/3830457968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf8c0039f0 msgr2=0x7fcf8c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:03.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa61cd640 1 --2- 172.21.15.62:0/3830457968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf8c0039f0 0x7fcf8c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.728+0000 7fcfa61cd640 1 -- 172.21.15.62:0/3830457968 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8c003530 con 0x7fcf8c004d50 2024-04-18T10:07:03.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa61cd640 1 --2- 172.21.15.62:0/3830457968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf8c004d50 0x7fcf8c0051d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fcfa009e6d0 tx=0x7fcf9801fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d3d753c26fb477 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9800a480 con 0x7fcf8c004d50 2024-04-18T10:07:03.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf980aa6b0 con 0x7fcf8c004d50 2024-04-18T10:07:03.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa49ca640 1 -- 172.21.15.62:0/3830457968 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf980aaaa0 con 0x7fcf8c004d50 2024-04-18T10:07:03.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 -- 172.21.15.62:0/3830457968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf8c004d50 msgr2=0x7fcf8c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 --2- 172.21.15.62:0/3830457968 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcf8c004d50 0x7fcf8c0051d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fcfa009e6d0 tx=0x7fcf9801fa60 comp rx=0 tx=0).stop 2024-04-18T10:07:03.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa61cd640 1 -- 172.21.15.62:0/3830457968 reap_dead start 2024-04-18T10:07:03.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 -- 172.21.15.62:0/3830457968 shutdown_connections 2024-04-18T10:07:03.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 -- 172.21.15.62:0/3830457968 >> 172.21.15.62:0/3830457968 conn(0x7fcfa00b9420 msgr2=0x7fcfa00b9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:03.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 -- 172.21.15.62:0/3830457968 shutdown_connections 2024-04-18T10:07:03.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 -- 172.21.15.62:0/3830457968 wait complete. 2024-04-18T10:07:03.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 Processor -- start 2024-04-18T10:07:03.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 -- start start 2024-04-18T10:07:03.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa013e570 0x7fcfa01dbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa01dc100 0x7fcfa00984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfa0098a30 0x7fcfa01def10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa009ba00 con 0x7fcfa01dc100 2024-04-18T10:07:03.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa01499c0 con 0x7fcfa0098a30 2024-04-18T10:07:03.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa009bb80 con 0x7fcfa013e570 2024-04-18T10:07:03.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa51cb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa01dc100 0x7fcfa00984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa51cb640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa01dc100 0x7fcfa00984f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35902/0 (socket says 172.21.15.62:35902) 2024-04-18T10:07:03.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa51cb640 1 -- 172.21.15.62:0/1714457127 learned_addr learned my addr 172.21.15.62:0/1714457127 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:03.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa59cc640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa013e570 0x7fcfa01dbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa61cd640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfa0098a30 0x7fcfa01def10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa51cb640 1 -- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa013e570 msgr2=0x7fcfa01dbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa51cb640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa013e570 0x7fcfa01dbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa51cb640 1 -- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfa0098a30 msgr2=0x7fcfa01def10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa51cb640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfa0098a30 0x7fcfa01def10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa51cb640 1 -- 172.21.15.62:0/1714457127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa01487e0 con 0x7fcfa01dc100 2024-04-18T10:07:03.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa61cd640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfa0098a30 0x7fcfa01def10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:03.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa51cb640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa01dc100 0x7fcfa00984f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000e4f0 tx=0x7fcf9000e9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcf967fc640 1 -- 172.21.15.62:0/1714457127 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf900052e0 con 0x7fcfa01dc100 2024-04-18T10:07:03.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcf967fc640 1 -- 172.21.15.62:0/1714457127 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf90018070 con 0x7fcfa01dc100 2024-04-18T10:07:03.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 -- 172.21.15.62:0/1714457127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa01dc580 con 0x7fcfa01dc100 2024-04-18T10:07:03.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcf967fc640 1 -- 172.21.15.62:0/1714457127 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90005480 con 0x7fcfa01dc100 2024-04-18T10:07:03.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.732+0000 7fcfa7c30640 1 -- 172.21.15.62:0/1714457127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa0148a50 con 0x7fcfa01dc100 2024-04-18T10:07:03.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.736+0000 7fcfa7c30640 1 -- 172.21.15.62:0/1714457127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa00694a0 con 0x7fcfa01dc100 2024-04-18T10:07:03.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.736+0000 7fcf967fc640 1 -- 172.21.15.62:0/1714457127 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcf90003820 con 0x7fcfa01dc100 2024-04-18T10:07:03.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.736+0000 7fcf967fc640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf84068380 0x7fcf8406a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:03.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.736+0000 7fcf967fc640 1 -- 172.21.15.62:0/1714457127 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fcf90086290 con 0x7fcfa01dc100 2024-04-18T10:07:03.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.736+0000 7fcfa59cc640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf84068380 0x7fcf8406a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:03.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.736+0000 7fcfa59cc640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf84068380 0x7fcf8406a840 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009a30 tx=0x7fcf9c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:03.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.740+0000 7fcf967fc640 1 -- 172.21.15.62:0/1714457127 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcfa01dc580 con 0x7fcfa01dc100 2024-04-18T10:07:03.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 -- 172.21.15.62:0/1714457127 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fcfa0068b20 con 0x7fcf84068380 2024-04-18T10:07:03.901 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcf967fc640 1 -- 172.21.15.62:0/1714457127 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fcfa0068b20 con 0x7fcf84068380 2024-04-18T10:07:03.901 INFO:teuthology.orchestra.run.smithi062.stderr:instructed osd(s) 7 to deep-scrub 2024-04-18T10:07:03.903 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 -- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf84068380 msgr2=0x7fcf8406a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf84068380 0x7fcf8406a840 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009a30 tx=0x7fcf9c008040 comp rx=0 tx=0).stop 2024-04-18T10:07:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 -- 172.21.15.62:0/1714457127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa01dc100 msgr2=0x7fcfa00984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa01dc100 0x7fcfa00984f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000e4f0 tx=0x7fcf9000e9b0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 -- 172.21.15.62:0/1714457127 shutdown_connections 2024-04-18T10:07:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fcf84068380 0x7fcf8406a840 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfa0098a30 0x7fcfa01def10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcfa01dc100 0x7fcfa00984f0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.904 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 --2- 172.21.15.62:0/1714457127 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfa013e570 0x7fcfa01dbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 -- 172.21.15.62:0/1714457127 >> 172.21.15.62:0/1714457127 conn(0x7fcfa00b9420 msgr2=0x7fcfa0144930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 -- 172.21.15.62:0/1714457127 shutdown_connections 2024-04-18T10:07:03.905 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:03.900+0000 7fcfa7c30640 1 -- 172.21.15.62:0/1714457127 wait complete. 2024-04-18T10:07:03.919 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T10:07:04.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09fcd0640 1 Processor -- start 2024-04-18T10:07:04.028 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09fcd0640 1 -- start start 2024-04-18T10:07:04.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09fcd0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0980b9d90 0x7fd09806d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:04.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09fcd0640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd098145f20 0x7fd0981504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:04.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09fcd0640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0980ba170 con 0x7fd09809be30 2024-04-18T10:07:04.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09fcd0640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fd09809cd30 con 0x7fd098145f20 2024-04-18T10:07:04.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09fcd0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd09814a060 con 0x7fd0980b9d90 2024-04-18T10:07:04.030 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09e26d640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fd098145f20 0x7fd0981504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:04.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09d26b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd09809be30 0x7fd09806d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:54048/0 (socket says 172.21.15.62:54048) 2024-04-18T10:07:04.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09d26b640 1 -- 172.21.15.62:0/140850822 learned_addr learned my addr 172.21.15.62:0/140850822 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-04-18T10:07:04.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09da6c640 1 --2- 172.21.15.62:0/140850822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0980b9d90 0x7fd09806d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:04.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09e26d640 1 -- 172.21.15.62:0/140850822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0980b9d90 msgr2=0x7fd09806d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:04.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09e26d640 1 --2- 172.21.15.62:0/140850822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0980b9d90 0x7fd09806d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:04.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09e26d640 1 -- 172.21.15.62:0/140850822 >> v1:172.21.15.62:6789/0 conn(0x7fd09809be30 legacy=0x7fd09806d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:04.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09ca6a640 1 -- 172.21.15.62:0/140850822 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1469500458 0 0) 0x7fd0980ba170 con 0x7fd09809be30 2024-04-18T10:07:04.031 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09e26d640 1 -- 172.21.15.62:0/140850822 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd098149860 con 0x7fd098145f20 2024-04-18T10:07:04.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.028+0000 7fd09e26d640 1 --2- 172.21.15.62:0/140850822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd098145f20 0x7fd0981504e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd090009d90 tx=0x7fd09001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6683b16c85dd8043 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:04.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09ca6a640 1 -- 172.21.15.62:0/140850822 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900b2070 con 0x7fd098145f20 2024-04-18T10:07:04.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09ca6a640 1 -- 172.21.15.62:0/140850822 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0900a57e0 con 0x7fd098145f20 2024-04-18T10:07:04.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09ca6a640 1 -- 172.21.15.62:0/140850822 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0900a5bb0 con 0x7fd098145f20 2024-04-18T10:07:04.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 -- 172.21.15.62:0/140850822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd098145f20 msgr2=0x7fd0981504e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:04.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 --2- 172.21.15.62:0/140850822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd098145f20 0x7fd0981504e0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd090009d90 tx=0x7fd09001c850 comp rx=0 tx=0).stop 2024-04-18T10:07:04.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 -- 172.21.15.62:0/140850822 shutdown_connections 2024-04-18T10:07:04.033 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 --2- 172.21.15.62:0/140850822 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd098145f20 0x7fd0981504e0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:04.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 --2- 172.21.15.62:0/140850822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0980b9d90 0x7fd09806d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:04.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 -- 172.21.15.62:0/140850822 >> 172.21.15.62:0/140850822 conn(0x7fd09806c0d0 msgr2=0x7fd0980b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:04.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 -- 172.21.15.62:0/140850822 shutdown_connections 2024-04-18T10:07:04.034 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 -- 172.21.15.62:0/140850822 wait complete. 2024-04-18T10:07:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 Processor -- start 2024-04-18T10:07:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 -- start start 2024-04-18T10:07:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd09809be30 0x7fd0980b2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:04.036 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0980b9d90 0x7fd0981de0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd098145f20 0x7fd0981e20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fd0980b3fa0 con 0x7fd0980b9d90 2024-04-18T10:07:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fd0980b3e20 con 0x7fd09809be30 2024-04-18T10:07:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fd09806de60 con 0x7fd098145f20 2024-04-18T10:07:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09d26b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0980b9d90 0x7fd0981de0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09d26b640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0980b9d90 0x7fd0981de0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:35918/0 (socket says 172.21.15.62:35918) 2024-04-18T10:07:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09d26b640 1 -- 172.21.15.62:0/392328204 learned_addr learned my addr 172.21.15.62:0/392328204 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:04.037 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09d26b640 1 -- 172.21.15.62:0/392328204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd098145f20 msgr2=0x7fd0981e20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09d26b640 1 --2- 172.21.15.62:0/392328204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd098145f20 0x7fd0981e20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09d26b640 1 -- 172.21.15.62:0/392328204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd09809be30 msgr2=0x7fd0980b2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09d26b640 1 --2- 172.21.15.62:0/392328204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd09809be30 0x7fd0980b2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09d26b640 1 -- 172.21.15.62:0/392328204 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09814b0a0 con 0x7fd0980b9d90 2024-04-18T10:07:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09d26b640 1 --2- 172.21.15.62:0/392328204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0980b9d90 0x7fd0981de0f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd0880011f0 tx=0x7fd08800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd08e7fc640 1 -- 172.21.15.62:0/392328204 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088018070 con 0x7fd0980b9d90 2024-04-18T10:07:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd08e7fc640 1 -- 172.21.15.62:0/392328204 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0880074e0 con 0x7fd0980b9d90 2024-04-18T10:07:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 -- 172.21.15.62:0/392328204 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09814b380 con 0x7fd0980b9d90 2024-04-18T10:07:04.038 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd08e7fc640 1 -- 172.21.15.62:0/392328204 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd088004a10 con 0x7fd0980b9d90 2024-04-18T10:07:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.032+0000 7fd09fcd0640 1 -- 172.21.15.62:0/392328204 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd098072a30 con 0x7fd0980b9d90 2024-04-18T10:07:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.036+0000 7fd09fcd0640 1 -- 172.21.15.62:0/392328204 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd098068b90 con 0x7fd0980b9d90 2024-04-18T10:07:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.036+0000 7fd08e7fc640 1 -- 172.21.15.62:0/392328204 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd08801b070 con 0x7fd0980b9d90 2024-04-18T10:07:04.039 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.036+0000 7fd08e7fc640 1 --2- 172.21.15.62:0/392328204 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd07c068380 0x7fd07c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:04.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.036+0000 7fd08e7fc640 1 -- 172.21.15.62:0/392328204 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fd088085940 con 0x7fd0980b9d90 2024-04-18T10:07:04.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.036+0000 7fd09da6c640 1 --2- 172.21.15.62:0/392328204 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd07c068380 0x7fd07c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:04.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.036+0000 7fd09da6c640 1 --2- 172.21.15.62:0/392328204 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd07c068380 0x7fd07c06a840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd0981449e0 tx=0x7fd094008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:04.041 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.040+0000 7fd08e7fc640 1 -- 172.21.15.62:0/392328204 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd09814b0a0 con 0x7fd0980b9d90 2024-04-18T10:07:04.196 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.196+0000 7fd09fcd0640 1 -- 172.21.15.62:0/392328204 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd098073740 con 0x7fd07c068380 2024-04-18T10:07:04.198 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.196+0000 7fd08e7fc640 1 -- 172.21.15.62:0/392328204 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13428 (secure 0 0 0) 0x7fd098073740 con 0x7fd07c068380 2024-04-18T10:07:04.198 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:07:04.199 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-18T10:07:04.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.200+0000 7fd09fcd0640 1 -- 172.21.15.62:0/392328204 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd07c068380 msgr2=0x7fd07c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:04.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.200+0000 7fd09fcd0640 1 --2- 172.21.15.62:0/392328204 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd07c068380 0x7fd07c06a840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd0981449e0 tx=0x7fd094008040 comp rx=0 tx=0).stop 2024-04-18T10:07:04.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.200+0000 7fd09fcd0640 1 -- 172.21.15.62:0/392328204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0980b9d90 msgr2=0x7fd0981de0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:04.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.200+0000 7fd09fcd0640 1 --2- 172.21.15.62:0/392328204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0980b9d90 0x7fd0981de0f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fd0880011f0 tx=0x7fd08800c9b0 comp rx=0 tx=0).stop 2024-04-18T10:07:04.200 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.200+0000 7fd09fcd0640 1 -- 172.21.15.62:0/392328204 shutdown_connections 2024-04-18T10:07:04.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.200+0000 7fd09fcd0640 1 --2- 172.21.15.62:0/392328204 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fd07c068380 0x7fd07c06a840 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:04.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.200+0000 7fd09fcd0640 1 --2- 172.21.15.62:0/392328204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd098145f20 0x7fd0981e20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:04.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.200+0000 7fd09fcd0640 1 --2- 172.21.15.62:0/392328204 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0980b9d90 0x7fd0981de0f0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:04.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.200+0000 7fd09fcd0640 1 --2- 172.21.15.62:0/392328204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd09809be30 0x7fd0980b2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:04.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.200+0000 7fd09fcd0640 1 -- 172.21.15.62:0/392328204 >> 172.21.15.62:0/392328204 conn(0x7fd09806c0d0 msgr2=0x7fd09809d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:04.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.200+0000 7fd09fcd0640 1 -- 172.21.15.62:0/392328204 shutdown_connections 2024-04-18T10:07:04.201 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:04.200+0000 7fd09fcd0640 1 -- 172.21.15.62:0/392328204 wait complete. 2024-04-18T10:07:04.215 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-04-18T10:07:03.146304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":220320,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754400,"statfs":{"total":773094113280,"available":772868505600,"internally_reserved":0,"allocated":4235264,"data_stored":2351392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10891,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-54,"num_write_kb":-26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002098"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":192,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-18T10:06:56.300560+0000","last_change":"2024-04-18T10:05:49.183850+0000","last_active":"2024-04-18T10:06:56.300560+0000","last_peered":"2024-04-18T10:06:56.300560+0000","last_clean":"2024-04-18T10:06:56.300560+0000","last_became_active":"2024-04-18T10:05:49.181471+0000","last_became_peered":"2024-04-18T10:05:49.181471+0000","last_unstale":"2024-04-18T10:06:56.300560+0000","last_undegraded":"2024-04-18T10:06:56.300560+0000","last_fullsized":"2024-04-18T10:06:56.300560+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-18T10:05:47.511486+0000","last_clean_scrub_stamp":"2024-04-18T10:05:47.511486+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:06:51.214547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":974848,"data_stored":736700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":974848,"data_stored":736700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T10:07:04.217 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-18T10:05:47.511486+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=10, tm_min=5, tm_sec=47, tm_wday=3, tm_yday=109, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=18, tm_hour=10, tm_min=6, tm_sec=59, tm_wday=3, tm_yday=109, tm_isdst=0) 2024-04-18T10:07:04.218 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-18T10:07:24.219 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-18T10:07:24.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39cd37f640 1 Processor -- start 2024-04-18T10:07:24.353 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39cd37f640 1 -- start start 2024-04-18T10:07:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39cd37f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f39c8067a70 0x7f39c80b5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39cd37f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f39c8147db0 0x7f39c8144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39cd37f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39c8067e50 con 0x7f39c8148240 2024-04-18T10:07:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39cd37f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f39c80af500 con 0x7f39c8067a70 2024-04-18T10:07:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39cd37f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f39c80b54b0 con 0x7f39c8147db0 2024-04-18T10:07:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39c6ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f39c8067a70 0x7f39c80b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39c6ffd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f39c8067a70 0x7f39c80b5900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44684/0 (socket says 172.21.15.62:44684) 2024-04-18T10:07:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39c6ffd640 1 -- 172.21.15.62:0/4248998813 learned_addr learned my addr 172.21.15.62:0/4248998813 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39c77fe640 1 --1- 172.21.15.62:0/4248998813 >> v1:172.21.15.62:6789/0 conn(0x7f39c8148240 0x7f39c814b320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:60208/0 (socket says 172.21.15.62:60208) 2024-04-18T10:07:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39c67fc640 1 --2- 172.21.15.62:0/4248998813 >> v2:172.21.15.145:3300/0 conn(0x7f39c8147db0 0x7f39c8144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39c6ffd640 1 -- 172.21.15.62:0/4248998813 >> v2:172.21.15.145:3300/0 conn(0x7f39c8147db0 msgr2=0x7f39c8144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39c6ffd640 1 --2- 172.21.15.62:0/4248998813 >> v2:172.21.15.145:3300/0 conn(0x7f39c8147db0 0x7f39c8144ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39c6ffd640 1 -- 172.21.15.62:0/4248998813 >> v1:172.21.15.62:6789/0 conn(0x7f39c8148240 legacy=0x7f39c814b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39c5ffb640 1 -- 172.21.15.62:0/4248998813 <== mon.0 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 246575499 0 0) 0x7f39c8067e50 con 0x7f39c8148240 2024-04-18T10:07:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39c6ffd640 1 -- 172.21.15.62:0/4248998813 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c80b5ee0 con 0x7f39c8067a70 2024-04-18T10:07:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.352+0000 7f39c67fc640 1 --2- 172.21.15.62:0/4248998813 >> v2:172.21.15.145:3300/0 conn(0x7f39c8147db0 0x7f39c8144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:07:24.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c6ffd640 1 --2- 172.21.15.62:0/4248998813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39c8067a70 0x7f39c80b5900 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f39c806d290 tx=0x7f39b801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3798afa5340532ab server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:24.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c5ffb640 1 -- 172.21.15.62:0/4248998813 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b80a5620 con 0x7f39c8067a70 2024-04-18T10:07:24.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c5ffb640 1 -- 172.21.15.62:0/4248998813 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39b80a57c0 con 0x7f39c8067a70 2024-04-18T10:07:24.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c5ffb640 1 -- 172.21.15.62:0/4248998813 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39b80a5b90 con 0x7f39c8067a70 2024-04-18T10:07:24.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 -- 172.21.15.62:0/4248998813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39c8067a70 msgr2=0x7f39c80b5900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.361 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 --2- 172.21.15.62:0/4248998813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39c8067a70 0x7f39c80b5900 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f39c806d290 tx=0x7f39b801c960 comp rx=0 tx=0).stop 2024-04-18T10:07:24.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 -- 172.21.15.62:0/4248998813 shutdown_connections 2024-04-18T10:07:24.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 --2- 172.21.15.62:0/4248998813 >> v2:172.21.15.145:3300/0 conn(0x7f39c8147db0 0x7f39c8144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 --2- 172.21.15.62:0/4248998813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39c8067a70 0x7f39c80b5900 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 -- 172.21.15.62:0/4248998813 >> 172.21.15.62:0/4248998813 conn(0x7f39c806c380 msgr2=0x7f39c80b9cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:24.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 -- 172.21.15.62:0/4248998813 shutdown_connections 2024-04-18T10:07:24.362 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 -- 172.21.15.62:0/4248998813 wait complete. 2024-04-18T10:07:24.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 Processor -- start 2024-04-18T10:07:24.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 -- start start 2024-04-18T10:07:24.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39c8067a70 0x7f39c8143e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39c8147db0 0x7f39c81428c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39c8148240 0x7f39c8141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f39c8155670 con 0x7f39c8148240 2024-04-18T10:07:24.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f39c8145830 con 0x7f39c8147db0 2024-04-18T10:07:24.365 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39cd37f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f39c81557f0 con 0x7f39c8067a70 2024-04-18T10:07:24.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39c8148240 0x7f39c8141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c77fe640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39c8148240 0x7f39c8141be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44686/0 (socket says 172.21.15.62:44686) 2024-04-18T10:07:24.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c77fe640 1 -- 172.21.15.62:0/1747926035 learned_addr learned my addr 172.21.15.62:0/1747926035 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:24.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c67fc640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39c8147db0 0x7f39c81428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c6ffd640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39c8067a70 0x7f39c8143e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c77fe640 1 -- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39c8067a70 msgr2=0x7f39c8143e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c77fe640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39c8067a70 0x7f39c8143e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.366 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c77fe640 1 -- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39c8147db0 msgr2=0x7f39c81428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c77fe640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39c8147db0 0x7f39c81428c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c77fe640 1 -- 172.21.15.62:0/1747926035 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39c81414e0 con 0x7f39c8148240 2024-04-18T10:07:24.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c6ffd640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39c8067a70 0x7f39c8143e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:24.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.360+0000 7f39c77fe640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39c8148240 0x7f39c8141be0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f39c000c470 tx=0x7f39c000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:24.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.364+0000 7f39bf7fe640 1 -- 172.21.15.62:0/1747926035 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0018070 con 0x7f39c8148240 2024-04-18T10:07:24.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.364+0000 7f39bf7fe640 1 -- 172.21.15.62:0/1747926035 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39c00074e0 con 0x7f39c8148240 2024-04-18T10:07:24.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.364+0000 7f39cd37f640 1 -- 172.21.15.62:0/1747926035 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39c8143410 con 0x7f39c8148240 2024-04-18T10:07:24.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.364+0000 7f39bf7fe640 1 -- 172.21.15.62:0/1747926035 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c0004a30 con 0x7f39c8148240 2024-04-18T10:07:24.367 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.364+0000 7f39cd37f640 1 -- 172.21.15.62:0/1747926035 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39c8142e00 con 0x7f39c8148240 2024-04-18T10:07:24.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.364+0000 7f39cd37f640 1 -- 172.21.15.62:0/1747926035 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39c8068b70 con 0x7f39c8148240 2024-04-18T10:07:24.368 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.364+0000 7f39bf7fe640 1 -- 172.21.15.62:0/1747926035 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f39c001b070 con 0x7f39c8148240 2024-04-18T10:07:24.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.364+0000 7f39bf7fe640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f39a8068380 0x7f39a806a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.364+0000 7f39bf7fe640 1 -- 172.21.15.62:0/1747926035 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f39c00858c0 con 0x7f39c8148240 2024-04-18T10:07:24.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.364+0000 7f39c6ffd640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f39a8068380 0x7f39a806a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.364+0000 7f39c6ffd640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f39a8068380 0x7f39a806a840 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f39b80ae000 tx=0x7f39b80a7070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:24.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.368+0000 7f39bf7fe640 1 -- 172.21.15.62:0/1747926035 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f39c8143410 con 0x7f39c8148240 2024-04-18T10:07:24.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.532+0000 7f39cd37f640 1 -- 172.21.15.62:0/1747926035 --> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f39c80b8d30 con 0x7f39a8068380 2024-04-18T10:07:24.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39bf7fe640 1 -- 172.21.15.62:0/1747926035 <== mgr.4103 v2:172.21.15.145:6832/4004518230 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13354 (secure 0 0 0) 0x7f39c80b8d30 con 0x7f39a8068380 2024-04-18T10:07:24.536 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:07:24.537 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-04-18T10:07:24.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39cd37f640 1 -- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f39a8068380 msgr2=0x7f39a806a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39cd37f640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f39a8068380 0x7f39a806a840 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f39b80ae000 tx=0x7f39b80a7070 comp rx=0 tx=0).stop 2024-04-18T10:07:24.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39cd37f640 1 -- 172.21.15.62:0/1747926035 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39c8148240 msgr2=0x7f39c8141be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39cd37f640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39c8148240 0x7f39c8141be0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f39c000c470 tx=0x7f39c000c930 comp rx=0 tx=0).stop 2024-04-18T10:07:24.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39cd37f640 1 -- 172.21.15.62:0/1747926035 shutdown_connections 2024-04-18T10:07:24.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39cd37f640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f39a8068380 0x7f39a806a840 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39cd37f640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39c8148240 0x7f39c8141be0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39cd37f640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39c8147db0 0x7f39c81428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39cd37f640 1 --2- 172.21.15.62:0/1747926035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39c8067a70 0x7f39c8143e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39cd37f640 1 -- 172.21.15.62:0/1747926035 >> 172.21.15.62:0/1747926035 conn(0x7f39c806c380 msgr2=0x7f39c806da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:24.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39cd37f640 1 -- 172.21.15.62:0/1747926035 shutdown_connections 2024-04-18T10:07:24.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.536+0000 7f39cd37f640 1 -- 172.21.15.62:0/1747926035 wait complete. 2024-04-18T10:07:24.556 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-04-18T10:07:23.151494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":220300,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754420,"statfs":{"total":773094113280,"available":772868526080,"internally_reserved":0,"allocated":4235264,"data_stored":2351392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003147"},"pg_stats":[{"pgid":"1.0","version":"33'119","reported_seq":203,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-04-18T10:07:03.947681+0000","last_change":"2024-04-18T10:07:03.947759+0000","last_active":"2024-04-18T10:07:03.947681+0000","last_peered":"2024-04-18T10:07:03.947681+0000","last_clean":"2024-04-18T10:07:03.947681+0000","last_became_active":"2024-04-18T10:05:49.181471+0000","last_became_peered":"2024-04-18T10:05:49.181471+0000","last_unstale":"2024-04-18T10:07:03.947681+0000","last_undegraded":"2024-04-18T10:07:03.947681+0000","last_fullsized":"2024-04-18T10:07:03.947681+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"33'119","last_scrub_stamp":"2024-04-18T10:07:03.947502+0000","last_deep_scrub":"33'119","last_deep_scrub_stamp":"2024-04-18T10:07:03.947502+0000","last_clean_scrub_stamp":"2024-04-18T10:07:03.947502+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T18:12:13.930257+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":974848,"data_stored":736700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27392,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":974848,"data_stored":736700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-18T10:07:24.557 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-18T10:07:24.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1868f640 1 Processor -- start 2024-04-18T10:07:24.677 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1868f640 1 -- start start 2024-04-18T10:07:24.678 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1868f640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0a100b3eb0 0x7f0a100bdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.679 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1868f640 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0a1014b190 0x7f0a10147ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.681 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1868f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a1014b620 con 0x7f0a1009b9d0 2024-04-18T10:07:24.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1868f640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7f0a100b4290 con 0x7f0a100b3eb0 2024-04-18T10:07:24.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1868f640 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 -- 0x7f0a1014c100 con 0x7f0a1014b190 2024-04-18T10:07:24.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1642b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0a100b3eb0 0x7f0a100bdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1642b640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7f0a100b3eb0 0x7f0a100bdd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44702/0 (socket says 172.21.15.62:44702) 2024-04-18T10:07:24.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1642b640 1 -- 172.21.15.62:0/2376416595 learned_addr learned my addr 172.21.15.62:0/2376416595 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:24.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1642b640 1 -- 172.21.15.62:0/2376416595 >> v2:172.21.15.145:3300/0 conn(0x7f0a1014b190 msgr2=0x7f0a10147ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:24.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a15c2a640 1 --2- 172.21.15.62:0/2376416595 >> v2:172.21.15.145:3300/0 conn(0x7f0a1014b190 0x7f0a10147ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.682 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1642b640 1 --2- 172.21.15.62:0/2376416595 >> v2:172.21.15.145:3300/0 conn(0x7f0a1014b190 0x7f0a10147ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1642b640 1 -- 172.21.15.62:0/2376416595 >> v1:172.21.15.145:6789/0 conn(0x7f0a1009b9d0 legacy=0x7f0a1006c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:24.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a1642b640 1 -- 172.21.15.62:0/2376416595 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1014b900 con 0x7f0a100b3eb0 2024-04-18T10:07:24.683 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.676+0000 7f0a15c2a640 1 --2- 172.21.15.62:0/2376416595 >> v2:172.21.15.145:3300/0 conn(0x7f0a1014b190 0x7f0a10147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:24.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a1642b640 1 --2- 172.21.15.62:0/2376416595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a100b3eb0 0x7f0a100bdd30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0a1009c230 tx=0x7f0a0401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d486720a3b422a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:24.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a15429640 1 -- 172.21.15.62:0/2376416595 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a040b0070 con 0x7f0a100b3eb0 2024-04-18T10:07:24.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a15429640 1 -- 172.21.15.62:0/2376416595 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a040ab600 con 0x7f0a100b3eb0 2024-04-18T10:07:24.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a15429640 1 -- 172.21.15.62:0/2376416595 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a040ab9d0 con 0x7f0a100b3eb0 2024-04-18T10:07:24.688 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a1868f640 1 -- 172.21.15.62:0/2376416595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a100b3eb0 msgr2=0x7f0a100bdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a1868f640 1 --2- 172.21.15.62:0/2376416595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a100b3eb0 0x7f0a100bdd30 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0a1009c230 tx=0x7f0a0401c450 comp rx=0 tx=0).stop 2024-04-18T10:07:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a1868f640 1 -- 172.21.15.62:0/2376416595 shutdown_connections 2024-04-18T10:07:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a1868f640 1 --2- 172.21.15.62:0/2376416595 >> v2:172.21.15.145:3300/0 conn(0x7f0a1014b190 0x7f0a10147ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a1868f640 1 --2- 172.21.15.62:0/2376416595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a100b3eb0 0x7f0a100bdd30 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a1868f640 1 -- 172.21.15.62:0/2376416595 >> 172.21.15.62:0/2376416595 conn(0x7f0a10067e10 msgr2=0x7f0a100bd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a1868f640 1 -- 172.21.15.62:0/2376416595 shutdown_connections 2024-04-18T10:07:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a1868f640 1 -- 172.21.15.62:0/2376416595 wait complete. 2024-04-18T10:07:24.689 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.680+0000 7f0a1868f640 1 Processor -- start 2024-04-18T10:07:24.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a1868f640 1 -- start start 2024-04-18T10:07:24.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a1868f640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a100b3eb0 0x7f0a101405a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a1868f640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a10142050 0x7f0a1013d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a1868f640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a1013d770 0x7f0a101e3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a1868f640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7f0a101558d0 con 0x7f0a1013d770 2024-04-18T10:07:24.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a1868f640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7f0a10155750 con 0x7f0a100b3eb0 2024-04-18T10:07:24.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a1868f640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7f0a10155a50 con 0x7f0a10142050 2024-04-18T10:07:24.690 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a16c2c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a1013d770 0x7f0a101e3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a16c2c640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a1013d770 0x7f0a101e3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44718/0 (socket says 172.21.15.62:44718) 2024-04-18T10:07:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a16c2c640 1 -- 172.21.15.62:0/812920417 learned_addr learned my addr 172.21.15.62:0/812920417 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a1642b640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a100b3eb0 0x7f0a101405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a15c2a640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a10142050 0x7f0a1013d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a16c2c640 1 -- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a10142050 msgr2=0x7f0a1013d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a16c2c640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a10142050 0x7f0a1013d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a16c2c640 1 -- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a100b3eb0 msgr2=0x7f0a101405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.691 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a16c2c640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a100b3eb0 0x7f0a101405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a16c2c640 1 -- 172.21.15.62:0/812920417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a1013dbf0 con 0x7f0a1013d770 2024-04-18T10:07:24.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a1642b640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a100b3eb0 0x7f0a101405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:07:24.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a15c2a640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a10142050 0x7f0a1013d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-18T10:07:24.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a16c2c640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a1013d770 0x7f0a101e3fd0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800c7b0 tx=0x7f0a0800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:24.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a02ffd640 1 -- 172.21.15.62:0/812920417 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08016020 con 0x7f0a1013d770 2024-04-18T10:07:24.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a02ffd640 1 -- 172.21.15.62:0/812920417 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a0800a9f0 con 0x7f0a1013d770 2024-04-18T10:07:24.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a1868f640 1 -- 172.21.15.62:0/812920417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a1013b450 con 0x7f0a1013d770 2024-04-18T10:07:24.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a02ffd640 1 -- 172.21.15.62:0/812920417 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a0800ade0 con 0x7f0a1013d770 2024-04-18T10:07:24.692 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a1868f640 1 -- 172.21.15.62:0/812920417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a1013b6f0 con 0x7f0a1013d770 2024-04-18T10:07:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a1868f640 1 -- 172.21.15.62:0/812920417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a10068c00 con 0x7f0a1013d770 2024-04-18T10:07:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a02ffd640 1 -- 172.21.15.62:0/812920417 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0a08019020 con 0x7f0a1013d770 2024-04-18T10:07:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a02ffd640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f09f40683b0 0x7f09f406a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.684+0000 7f0a02ffd640 1 -- 172.21.15.62:0/812920417 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7f0a08086300 con 0x7f0a1013d770 2024-04-18T10:07:24.693 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.688+0000 7f0a1642b640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f09f40683b0 0x7f09f406a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.688+0000 7f0a1642b640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f09f40683b0 0x7f09f406a870 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0a040ad000 tx=0x7f0a040a6000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:24.694 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.692+0000 7f0a02ffd640 1 -- 172.21.15.62:0/812920417 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0a08020070 con 0x7f0a1013d770 2024-04-18T10:07:24.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.848+0000 7f0a1868f640 1 -- 172.21.15.62:0/812920417 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f0a1009b2d0 con 0x7f0a1013d770 2024-04-18T10:07:24.854 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.852+0000 7f0a02ffd640 1 -- 172.21.15.62:0/812920417 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v12)=0 v12) ==== 123+0+0 (secure 0 0 0) 0x7f0a1013dbf0 con 0x7f0a1013d770 2024-04-18T10:07:24.856 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a02ffd640 1 -- 172.21.15.62:0/812920417 <== mon.0 v2:172.21.15.62:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0a08056bc0 con 0x7f0a1013d770 2024-04-18T10:07:24.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a1868f640 1 -- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f09f40683b0 msgr2=0x7f09f406a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a1868f640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f09f40683b0 0x7f09f406a870 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0a040ad000 tx=0x7f0a040a6000 comp rx=0 tx=0).stop 2024-04-18T10:07:24.858 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a1868f640 1 -- 172.21.15.62:0/812920417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a1013d770 msgr2=0x7f0a101e3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a1868f640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a1013d770 0x7f0a101e3fd0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800c7b0 tx=0x7f0a0800cc70 comp rx=0 tx=0).stop 2024-04-18T10:07:24.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a1868f640 1 -- 172.21.15.62:0/812920417 shutdown_connections 2024-04-18T10:07:24.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a1868f640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7f09f40683b0 0x7f09f406a870 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a1868f640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a1013d770 0x7f0a101e3fd0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a1868f640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a10142050 0x7f0a1013d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a1868f640 1 --2- 172.21.15.62:0/812920417 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a100b3eb0 0x7f0a101405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a1868f640 1 -- 172.21.15.62:0/812920417 >> 172.21.15.62:0/812920417 conn(0x7f0a10067e10 msgr2=0x7f0a100ba810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:24.859 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a1868f640 1 -- 172.21.15.62:0/812920417 shutdown_connections 2024-04-18T10:07:24.860 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.856+0000 7f0a1868f640 1 -- 172.21.15.62:0/812920417 wait complete. 2024-04-18T10:07:24.873 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-18T10:07:24.874 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-18T10:07:24.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa82dabd640 1 Processor -- start 2024-04-18T10:07:24.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa82dabd640 1 -- start start 2024-04-18T10:07:24.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa82dabd640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa8280b9d90 0x7fa8281450e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa82dabd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8280b4970 0x7fa828148840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa82dabd640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa828145620 con 0x7fa8280b44e0 2024-04-18T10:07:24.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa82dabd640 1 -- --> v2:172.21.15.62:3300/0 -- mon_getmap magic: 0 -- 0x7fa828145b80 con 0x7fa8280b9d90 2024-04-18T10:07:24.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa82dabd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fa8280ba170 con 0x7fa8280b4970 2024-04-18T10:07:24.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa8277fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa8280b9d90 0x7fa8281450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa8277fe640 1 --2- >> v2:172.21.15.62:3300/0 conn(0x7fa8280b9d90 0x7fa8281450e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44734/0 (socket says 172.21.15.62:44734) 2024-04-18T10:07:24.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa8277fe640 1 -- 172.21.15.62:0/390150446 learned_addr learned my addr 172.21.15.62:0/390150446 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:24.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa827fff640 1 --2- 172.21.15.62:0/390150446 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8280b4970 0x7fa828148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa8277fe640 1 -- 172.21.15.62:0/390150446 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8280b4970 msgr2=0x7fa828148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:24.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa8277fe640 1 --2- 172.21.15.62:0/390150446 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8280b4970 0x7fa828148840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa8277fe640 1 -- 172.21.15.62:0/390150446 >> v1:172.21.15.145:6789/0 conn(0x7fa8280b44e0 legacy=0x7fa828148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-18T10:07:24.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa8277fe640 1 -- 172.21.15.62:0/390150446 --> v2:172.21.15.62:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8281458a0 con 0x7fa8280b9d90 2024-04-18T10:07:24.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa827fff640 1 --2- 172.21.15.62:0/390150446 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8280b4970 0x7fa828148840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:24.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa8277fe640 1 --2- 172.21.15.62:0/390150446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8280b9d90 0x7fa8281450e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa81800bb40 tx=0x7fa81801e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ae29860f9b5ed3 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:24.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa826ffd640 1 -- 172.21.15.62:0/390150446 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8180b4070 con 0x7fa8280b9d90 2024-04-18T10:07:24.988 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa826ffd640 1 -- 172.21.15.62:0/390150446 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa8180a7780 con 0x7fa8280b9d90 2024-04-18T10:07:24.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.984+0000 7fa826ffd640 1 -- 172.21.15.62:0/390150446 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8180a7b70 con 0x7fa8280b9d90 2024-04-18T10:07:24.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 -- 172.21.15.62:0/390150446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8280b9d90 msgr2=0x7fa8281450e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 --2- 172.21.15.62:0/390150446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8280b9d90 0x7fa8281450e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa81800bb40 tx=0x7fa81801e9f0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 -- 172.21.15.62:0/390150446 shutdown_connections 2024-04-18T10:07:24.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 --2- 172.21.15.62:0/390150446 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8280b4970 0x7fa828148840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 --2- 172.21.15.62:0/390150446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8280b9d90 0x7fa8281450e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 -- 172.21.15.62:0/390150446 >> 172.21.15.62:0/390150446 conn(0x7fa82806c0d0 msgr2=0x7fa8280b9420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:24.989 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 -- 172.21.15.62:0/390150446 shutdown_connections 2024-04-18T10:07:24.990 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 -- 172.21.15.62:0/390150446 wait complete. 2024-04-18T10:07:24.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 Processor -- start 2024-04-18T10:07:24.992 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 -- start start 2024-04-18T10:07:24.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8280b44e0 0x7fa82808d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8280b4970 0x7fa8281e05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8280b9d90 0x7fa8281e0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 -- 0x7fa828148f00 con 0x7fa8280b4970 2024-04-18T10:07:24.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 -- 0x7fa828148d80 con 0x7fa8280b44e0 2024-04-18T10:07:24.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 -- 0x7fa82806da20 con 0x7fa8280b9d90 2024-04-18T10:07:24.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa81ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8280b4970 0x7fa8281e05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.993 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa81ffff640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8280b4970 0x7fa8281e05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:44736/0 (socket says 172.21.15.62:44736) 2024-04-18T10:07:24.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa81ffff640 1 -- 172.21.15.62:0/258312399 learned_addr learned my addr 172.21.15.62:0/258312399 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-04-18T10:07:24.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa8277fe640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8280b44e0 0x7fa82808d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa827fff640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8280b9d90 0x7fa8281e0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa81ffff640 1 -- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8280b9d90 msgr2=0x7fa8281e0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa81ffff640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8280b9d90 0x7fa8281e0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.994 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa81ffff640 1 -- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8280b44e0 msgr2=0x7fa82808d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:24.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa81ffff640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8280b44e0 0x7fa82808d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:24.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa81ffff640 1 -- 172.21.15.62:0/258312399 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82808cdd0 con 0x7fa8280b4970 2024-04-18T10:07:24.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa8277fe640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8280b44e0 0x7fa82808d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:24.995 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa827fff640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8280b9d90 0x7fa8281e0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-18T10:07:24.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa81ffff640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8280b4970 0x7fa8281e05c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa81000c450 tx=0x7fa81000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:24.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa824ff9640 1 -- 172.21.15.62:0/258312399 <== mon.0 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810012830 con 0x7fa8280b4970 2024-04-18T10:07:24.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa824ff9640 1 -- 172.21.15.62:0/258312399 <== mon.0 v2:172.21.15.62:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa810017070 con 0x7fa8280b4970 2024-04-18T10:07:24.996 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 -- 172.21.15.62:0/258312399 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82808ed20 con 0x7fa8280b4970 2024-04-18T10:07:24.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa824ff9640 1 -- 172.21.15.62:0/258312399 <== mon.0 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa810012c50 con 0x7fa8280b4970 2024-04-18T10:07:24.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 -- 172.21.15.62:0/258312399 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa82808dbe0 con 0x7fa8280b4970 2024-04-18T10:07:24.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.988+0000 7fa82dabd640 1 -- 172.21.15.62:0/258312399 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa828068b90 con 0x7fa8280b4970 2024-04-18T10:07:24.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.992+0000 7fa824ff9640 1 -- 172.21.15.62:0/258312399 <== mon.0 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa81001c460 con 0x7fa8280b4970 2024-04-18T10:07:24.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.992+0000 7fa824ff9640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa8080683b0 0x7fa80806a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-18T10:07:24.997 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.992+0000 7fa824ff9640 1 -- 172.21.15.62:0/258312399 <== mon.0 v2:172.21.15.62:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4943+0+0 (secure 0 0 0) 0x7fa810086400 con 0x7fa8280b4970 2024-04-18T10:07:24.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.992+0000 7fa8277fe640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa8080683b0 0x7fa80806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-18T10:07:24.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.992+0000 7fa8277fe640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa8080683b0 0x7fa80806a870 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa8180b1000 tx=0x7fa818008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-18T10:07:24.999 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:24.996+0000 7fa824ff9640 1 -- 172.21.15.62:0/258312399 <== mon.0 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa810056cc0 con 0x7fa8280b4970 2024-04-18T10:07:25.191 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.188+0000 7fa82dabd640 1 -- 172.21.15.62:0/258312399 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa828146bb0 con 0x7fa8280b4970 2024-04-18T10:07:25.192 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.188+0000 7fa824ff9640 1 -- 172.21.15.62:0/258312399 <== mon.0 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+2222 (secure 0 0 0) 0x7fa8100565c0 con 0x7fa8280b4970 2024-04-18T10:07:25.192 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-04-18T10:07:25.192 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4705,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1649467687","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1649467687},{"type":"v1","addr":"172.21.15.145:6835","nonce":1649467687}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5241,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/1087980664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":1087980664},{"type":"v1","addr":"172.21.15.62:6833","nonce":1087980664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[]} 2024-04-18T10:07:25.193 INFO:teuthology.orchestra.run.smithi062.stderr:dumped fsmap epoch 18 2024-04-18T10:07:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.192+0000 7fa82dabd640 1 -- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa8080683b0 msgr2=0x7fa80806a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.192+0000 7fa82dabd640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa8080683b0 0x7fa80806a870 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa8180b1000 tx=0x7fa818008040 comp rx=0 tx=0).stop 2024-04-18T10:07:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.192+0000 7fa82dabd640 1 -- 172.21.15.62:0/258312399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8280b4970 msgr2=0x7fa8281e05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-18T10:07:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.192+0000 7fa82dabd640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8280b4970 0x7fa8281e05c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa81000c450 tx=0x7fa81000c910 comp rx=0 tx=0).stop 2024-04-18T10:07:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.192+0000 7fa82dabd640 1 -- 172.21.15.62:0/258312399 shutdown_connections 2024-04-18T10:07:25.194 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.192+0000 7fa82dabd640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:6832/4004518230,v1:172.21.15.145:6833/4004518230] conn(0x7fa8080683b0 0x7fa80806a870 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.192+0000 7fa82dabd640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8280b9d90 0x7fa8281e0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.192+0000 7fa82dabd640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa8280b4970 0x7fa8281e05c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.192+0000 7fa82dabd640 1 --2- 172.21.15.62:0/258312399 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8280b44e0 0x7fa82808d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-18T10:07:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.192+0000 7fa82dabd640 1 -- 172.21.15.62:0/258312399 >> 172.21.15.62:0/258312399 conn(0x7fa82806c0d0 msgr2=0x7fa8280b6570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-18T10:07:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.192+0000 7fa82dabd640 1 -- 172.21.15.62:0/258312399 shutdown_connections 2024-04-18T10:07:25.195 INFO:teuthology.orchestra.run.smithi062.stderr:2024-04-18T10:07:25.192+0000 7fa82dabd640 1 -- 172.21.15.62:0/258312399 wait complete. 2024-04-18T10:07:25.209 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-18T10:07:25.210 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-18T10:07:25.210 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-18T10:07:25.210 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:25.221 INFO:tasks.ceph.mds.a:Stopped 2024-04-18T10:07:25.221 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-18T10:07:25.222 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:25.234 INFO:tasks.ceph.mds.b:Stopped 2024-04-18T10:07:25.235 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-18T10:07:25.235 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-18T10:07:25.235 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:29.071 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-18T10:07:29.074 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-18T10:07:29.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-18T10:07:31.337 INFO:tasks.ceph.osd.0:Stopped 2024-04-18T10:07:31.338 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-18T10:07:31.338 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:31.399 INFO:tasks.ceph.osd.1:Stopped 2024-04-18T10:07:31.400 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-18T10:07:31.400 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:31.485 INFO:tasks.ceph.osd.2:Stopped 2024-04-18T10:07:31.485 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-18T10:07:31.485 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:31.578 INFO:tasks.ceph.osd.3:Stopped 2024-04-18T10:07:31.578 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-18T10:07:31.578 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:31.656 INFO:tasks.ceph.osd.4:Stopped 2024-04-18T10:07:31.656 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-18T10:07:31.657 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:31.730 INFO:tasks.ceph.osd.5:Stopped 2024-04-18T10:07:31.731 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-18T10:07:31.731 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:31.818 INFO:tasks.ceph.osd.6:Stopped 2024-04-18T10:07:31.819 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-18T10:07:31.819 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:31.918 INFO:tasks.ceph.osd.7:Stopped 2024-04-18T10:07:31.918 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-18T10:07:31.919 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-18T10:07:31.919 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:31.947 INFO:tasks.ceph.mgr.y:Stopped 2024-04-18T10:07:31.947 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-18T10:07:31.947 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:31.977 INFO:tasks.ceph.mgr.x:Stopped 2024-04-18T10:07:31.977 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-18T10:07:31.977 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-18T10:07:31.977 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:32.001 INFO:tasks.ceph.mon.a:Stopped 2024-04-18T10:07:32.001 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-18T10:07:32.001 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:32.032 INFO:tasks.ceph.mon.b:Stopped 2024-04-18T10:07:32.032 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-18T10:07:32.032 INFO:teuthology.orchestra.run:waiting for 300 2024-04-18T10:07:32.099 INFO:tasks.ceph.mon.c:Stopped 2024-04-18T10:07:32.099 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-18T10:07:32.100 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-18T10:07:32.118 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T10:07:32.118 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-18T10:07:32.580 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T10:07:32.580 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-18T10:07:32.827 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T10:07:32.828 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-18T10:07:33.017 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T10:07:33.018 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-18T10:07:33.211 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi145.front.sepia.ceph.com 2024-04-18T10:07:33.212 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-18T10:07:33.502 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi145.front.sepia.ceph.com 2024-04-18T10:07:33.503 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-18T10:07:33.754 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi145.front.sepia.ceph.com 2024-04-18T10:07:33.755 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-18T10:07:33.931 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi145.front.sepia.ceph.com 2024-04-18T10:07:33.931 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-18T10:07:34.112 INFO:tasks.ceph:Archiving mon data... 2024-04-18T10:07:34.113 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928/data/mon.a.tgz 2024-04-18T10:07:34.113 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-04-18T10:07:34.119 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.qY4loQPDi0 2024-04-18T10:07:34.120 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.qY4loQPDi0 2024-04-18T10:07:34.239 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.qY4loQPDi0 2024-04-18T10:07:34.307 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.qY4loQPDi0 is 268KB 2024-04-18T10:07:34.366 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.qY4loQPDi0 2024-04-18T10:07:34.372 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928/data/mon.b.tgz 2024-04-18T10:07:34.372 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-04-18T10:07:34.378 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.CVr3l5eIeP 2024-04-18T10:07:34.378 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.CVr3l5eIeP 2024-04-18T10:07:34.508 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.CVr3l5eIeP 2024-04-18T10:07:34.577 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.CVr3l5eIeP is 364KB 2024-04-18T10:07:34.638 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.CVr3l5eIeP 2024-04-18T10:07:34.644 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928/data/mon.c.tgz 2024-04-18T10:07:34.644 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-04-18T10:07:34.694 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.iwKZVMp8Ok 2024-04-18T10:07:34.694 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.iwKZVMp8Ok 2024-04-18T10:07:34.823 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.iwKZVMp8Ok 2024-04-18T10:07:34.834 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-18T10:07:34.889 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.iwKZVMp8Ok is 364KB 2024-04-18T10:07:34.950 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.iwKZVMp8Ok 2024-04-18T10:07:34.956 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-18T10:07:34.957 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-18T10:07:34.959 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-18T10:07:35.018 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-18T10:07:35.061 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-18T10:07:35.061 INFO:tasks.ceph:Archiving crash dumps... 2024-04-18T10:07:35.063 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928/remote/smithi062/crash 2024-04-18T10:07:35.064 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-18T10:07:35.078 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928/remote/smithi145/crash 2024-04-18T10:07:35.079 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-18T10:07:35.116 INFO:tasks.ceph:Shutting down logrotate 2024-04-18T10:07:35.117 DEBUG:teuthology.orchestra.run.smithi062:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-18T10:07:35.127 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-18T10:07:35.201 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-18T10:07:35.214 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-18T10:07:35.256 INFO:tasks.ceph:Compressing logs... 2024-04-18T10:07:35.256 DEBUG:teuthology.orchestra.run.smithi062:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-18T10:07:35.259 DEBUG:teuthology.orchestra.run.smithi145:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-18T10:07:35.282 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36331.log 2024-04-18T10:07:35.284 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33088.log 2024-04-18T10:07:35.284 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37581.log 2024-04-18T10:07:35.285 INFO:teuthology.orchestra.run.smithi062.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36331.log.gz 2024-04-18T10:07:35.285 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34747.log 2024-04-18T10:07:35.285 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33088.log: gzip/var/log/ceph/ceph-client.admin.37581.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35924.log 2024-04-18T10:07:35.285 INFO:teuthology.orchestra.run.smithi062.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33088.log.gz 2024-04-18T10:07:35.289 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34747.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37581.log.gz 2024-04-18T10:07:35.289 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33311.log 2024-04-18T10:07:35.289 INFO:teuthology.orchestra.run.smithi062.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34747.log.gz 2024-04-18T10:07:35.289 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-18T10:07:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35924.log: /var/log/ceph/ceph-client.admin.33311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34590.log 2024-04-18T10:07:35.290 INFO:teuthology.orchestra.run.smithi062.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33311.log.gz 2024-04-18T10:07:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33832.log 2024-04-18T10:07:35.290 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35924.log.gz 2024-04-18T10:07:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.0.34590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33831.log 2024-04-18T10:07:35.290 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33832.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-18T10:07:35.291 INFO:teuthology.orchestra.run.smithi062.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.0.34590.log.gz 2024-04-18T10:07:35.291 INFO:teuthology.orchestra.run.smithi062.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33832.log.gz 2024-04-18T10:07:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36462.log 2024-04-18T10:07:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.0.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33831.log.gz 2024-04-18T10:07:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36707.log 2024-04-18T10:07:35.291 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34173.log 2024-04-18T10:07:35.292 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36707.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36462.log.gz 2024-04-18T10:07:35.292 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36841.log 2024-04-18T10:07:35.303 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34877.log 2024-04-18T10:07:35.303 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34173.log: /var/log/ceph/ceph-client.admin.36841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36080.log 2024-04-18T10:07:35.304 INFO:teuthology.orchestra.run.smithi062.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36707.log.gz 2024-04-18T10:07:35.304 INFO:teuthology.orchestra.run.smithi062.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-18T10:07:35.304 INFO:teuthology.orchestra.run.smithi062.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34173.log.gz 2024-04-18T10:07:35.304 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34877.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.33062.log 2024-04-18T10:07:35.304 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36841.log.gz 2024-04-18T10:07:35.304 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36080.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34877.log.gz 2024-04-18T10:07:35.305 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36815.log 2024-04-18T10:07:35.305 INFO:teuthology.orchestra.run.smithi062.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36080.log.gz 2024-04-18T10:07:35.305 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34303.log 2024-04-18T10:07:35.305 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33062.log.gz 2024-04-18T10:07:35.305 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37815.log 2024-04-18T10:07:35.305 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34825.log 2024-04-18T10:07:35.305 INFO:teuthology.orchestra.run.smithi062.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36815.log.gz 2024-04-18T10:07:35.306 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34303.log.gz 2024-04-18T10:07:35.306 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35672.log 2024-04-18T10:07:35.306 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34825.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37815.log.gz 2024-04-18T10:07:35.306 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33837.log 2024-04-18T10:07:35.306 INFO:teuthology.orchestra.run.smithi062.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34825.log.gz 2024-04-18T10:07:35.306 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27718.log/var/log/ceph/ceph-client.admin.35672.log: 2024-04-18T10:07:35.306 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35516.log 2024-04-18T10:07:35.307 INFO:teuthology.orchestra.run.smithi062.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35672.log.gz 2024-04-18T10:07:35.307 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27718.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33837.log.gz 2024-04-18T10:07:35.307 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35248.log 2024-04-18T10:07:35.307 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27718.log.gz 2024-04-18T10:07:35.307 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33549.log 2024-04-18T10:07:35.308 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36950.log 2024-04-18T10:07:35.308 INFO:teuthology.orchestra.run.smithi062.stderr: 89.4%/var/log/ceph/ceph-client.admin.35248.log: -- replaced with /var/log/ceph/ceph-client.admin.35516.log.gz 2024-04-18T10:07:35.308 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36184.log 2024-04-18T10:07:35.308 INFO:teuthology.orchestra.run.smithi062.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35248.log.gz 2024-04-18T10:07:35.308 INFO:teuthology.orchestra.run.smithi062.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33549.log.gz 2024-04-18T10:07:35.308 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34981.log 2024-04-18T10:07:35.308 INFO:teuthology.orchestra.run.smithi062.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.36950.log.gz 2024-04-18T10:07:35.309 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34225.log 2024-04-18T10:07:35.309 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34981.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36184.log.gz 2024-04-18T10:07:35.309 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36872.log 2024-04-18T10:07:35.309 INFO:teuthology.orchestra.run.smithi062.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34981.log.gz 2024-04-18T10:07:35.309 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35137.log 2024-04-18T10:07:35.309 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34955.log 2024-04-18T10:07:35.309 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34225.log.gz 2024-04-18T10:07:35.310 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35137.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36872.log.gz 2024-04-18T10:07:35.310 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34773.log 2024-04-18T10:07:35.310 INFO:teuthology.orchestra.run.smithi062.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35137.log.gz 2024-04-18T10:07:35.310 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37028.log 2024-04-18T10:07:35.310 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34773.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-18T10:07:35.310 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34955.log.gz 2024-04-18T10:07:35.310 INFO:teuthology.orchestra.run.smithi062.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34773.log.gz 2024-04-18T10:07:35.311 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36488.log 2024-04-18T10:07:35.311 INFO:teuthology.orchestra.run.smithi062.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37028.log.gz 2024-04-18T10:07:35.311 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34643.log 2024-04-18T10:07:35.311 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34355.log 2024-04-18T10:07:35.311 INFO:teuthology.orchestra.run.smithi062.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36488.log.gz 2024-04-18T10:07:35.311 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36514.log 2024-04-18T10:07:35.311 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34355.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34643.log.gz 2024-04-18T10:07:35.311 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37345.log 2024-04-18T10:07:35.312 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34355.log.gz 2024-04-18T10:07:35.312 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33627.log 2024-04-18T10:07:35.312 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36514.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.37345.log: -- /var/log/ceph/ceph-mon.a.log 2024-04-18T10:07:35.312 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36514.log.gz 2024-04-18T10:07:35.312 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32986.log 2024-04-18T10:07:35.312 INFO:teuthology.orchestra.run.smithi062.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37345.log.gz 2024-04-18T10:07:35.312 INFO:teuthology.orchestra.run.smithi062.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33627.log.gz 2024-04-18T10:07:35.312 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33629.log 2024-04-18T10:07:35.313 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.32986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33497.log 2024-04-18T10:07:35.336 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27616.log 2024-04-18T10:07:35.357 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27736.log 2024-04-18T10:07:35.357 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.27616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27712.log 2024-04-18T10:07:35.357 INFO:teuthology.orchestra.run.smithi145.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27616.log.gz 2024-04-18T10:07:35.357 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.27736.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-18T10:07:35.357 INFO:teuthology.orchestra.run.smithi145.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.27736.log.gz 2024-04-18T10:07:35.358 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.27712.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-18T10:07:35.358 INFO:teuthology.orchestra.run.smithi145.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.27712.log.gz 2024-04-18T10:07:35.358 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-18T10:07:35.358 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27640.log 2024-04-18T10:07:35.358 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27688.log 2024-04-18T10:07:35.358 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.27640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27784.log 2024-04-18T10:07:35.359 INFO:teuthology.orchestra.run.smithi145.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27640.log.gz 2024-04-18T10:07:35.359 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.27688.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-18T10:07:35.359 INFO:teuthology.orchestra.run.smithi145.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27688.log.gz 2024-04-18T10:07:35.359 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27664.log 2024-04-18T10:07:35.359 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.27784.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27760.log 2024-04-18T10:07:35.359 INFO:teuthology.orchestra.run.smithi145.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27784.log.gz 2024-04-18T10:07:35.359 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.27664.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-18T10:07:35.360 INFO:teuthology.orchestra.run.smithi145.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27664.log.gz 2024-04-18T10:07:35.360 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.27760.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-18T10:07:35.360 INFO:teuthology.orchestra.run.smithi145.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27760.log.gz 2024-04-18T10:07:35.360 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-18T10:07:35.360 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-18T10:07:35.360 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-18T10:07:35.360 INFO:teuthology.orchestra.run.smithi145.stderr: 91.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-18T10:07:35.361 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-18T10:07:35.361 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-mon.b.log: 91.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-18T10:07:35.361 INFO:teuthology.orchestra.run.smithi145.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-18T10:07:35.387 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-osd.4.log: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-18T10:07:35.394 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33140.log 2024-04-18T10:07:35.394 INFO:teuthology.orchestra.run.smithi062.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32986.log.gz 2024-04-18T10:07:35.394 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.33497.log: /var/log/ceph/ceph-client.admin.36590.log 2024-04-18T10:07:35.395 INFO:teuthology.orchestra.run.smithi062.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33629.log.gz 2024-04-18T10:07:35.395 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36924.log 2024-04-18T10:07:35.395 INFO:teuthology.orchestra.run.smithi062.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33497.log.gz 2024-04-18T10:07:35.395 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33140.log.gz 2024-04-18T10:07:35.395 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35355.log 2024-04-18T10:07:35.395 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36924.log: gzip -5 89.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.36590.log.gz /var/log/ceph/ceph-client.admin.36158.log 2024-04-18T10:07:35.395 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.396 INFO:teuthology.orchestra.run.smithi062.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36924.log.gz 2024-04-18T10:07:35.396 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37425.log 2024-04-18T10:07:35.396 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36158.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35355.log.gz 2024-04-18T10:07:35.396 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33274.log 2024-04-18T10:07:35.396 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36158.log.gz 2024-04-18T10:07:35.396 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34903.log 2024-04-18T10:07:35.396 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36275.log 2024-04-18T10:07:35.397 INFO:teuthology.orchestra.run.smithi062.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37425.log.gz 2024-04-18T10:07:35.397 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34903.log: 89.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.33274.log.gz -5 2024-04-18T10:07:35.397 INFO:teuthology.orchestra.run.smithi062.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36383.log 2024-04-18T10:07:35.397 INFO:teuthology.orchestra.run.smithi062.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34903.log.gz 2024-04-18T10:07:35.397 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37711.log 2024-04-18T10:07:35.397 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36275.log.gz 2024-04-18T10:07:35.397 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32930.log 2024-04-18T10:07:35.397 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37711.log: gzip 88.7% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.36383.log.gz /var/log/ceph/ceph-client.admin.35381.log 2024-04-18T10:07:35.398 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.398 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.32930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27620.log 2024-04-18T10:07:35.398 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37711.log.gz 2024-04-18T10:07:35.398 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35381.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.32930.log.gz 2024-04-18T10:07:35.398 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33393.log 2024-04-18T10:07:35.398 INFO:teuthology.orchestra.run.smithi062.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35381.log.gz 2024-04-18T10:07:35.398 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35542.log 2024-04-18T10:07:35.399 INFO:teuthology.orchestra.run.smithi062.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.27620.log.gz 2024-04-18T10:07:35.399 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33419.log 2024-04-18T10:07:35.399 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33393.log: /var/log/ceph/ceph-client.admin.35542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35698.log 2024-04-18T10:07:35.399 INFO:teuthology.orchestra.run.smithi062.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33393.log.gz 2024-04-18T10:07:35.399 INFO:teuthology.orchestra.run.smithi062.stderr: 89.5%/var/log/ceph/ceph-client.admin.33419.log: -- replaced with /var/log/ceph/ceph-client.admin.35542.log.gz 2024-04-18T10:07:35.399 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34462.log 2024-04-18T10:07:35.399 INFO:teuthology.orchestra.run.smithi062.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33419.log.gz 2024-04-18T10:07:35.400 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36646.log 2024-04-18T10:07:35.400 INFO:teuthology.orchestra.run.smithi062.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35698.log.gz 2024-04-18T10:07:35.400 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37080.log 2024-04-18T10:07:35.400 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.0.36646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36132.log 2024-04-18T10:07:35.400 INFO:teuthology.orchestra.run.smithi062.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34462.log.gz 2024-04-18T10:07:35.400 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37080.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-18T10:07:35.400 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36132.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37080.log.gz 2024-04-18T10:07:35.400 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36357.log 2024-04-18T10:07:35.401 INFO:teuthology.orchestra.run.smithi062.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36132.log.gz 2024-04-18T10:07:35.401 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35433.log 2024-04-18T10:07:35.401 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37054.log 2024-04-18T10:07:35.401 INFO:teuthology.orchestra.run.smithi062.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36357.log.gz/var/log/ceph/ceph-client.admin.35433.log: 2024-04-18T10:07:35.401 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33114.log 2024-04-18T10:07:35.401 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35433.log.gz 2024-04-18T10:07:35.401 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35878.log 2024-04-18T10:07:35.401 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37054.log: /var/log/ceph/ceph-client.admin.33114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27499.log 2024-04-18T10:07:35.402 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35878.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33114.log.gz 2024-04-18T10:07:35.402 INFO:teuthology.orchestra.run.smithi062.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37054.log.gz 2024-04-18T10:07:35.402 INFO:teuthology.orchestra.run.smithi062.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.0.36646.log.gz 2024-04-18T10:07:35.402 INFO:teuthology.orchestra.run.smithi062.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35878.log.gz 2024-04-18T10:07:35.402 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33621.log 2024-04-18T10:07:35.402 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.27499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35490.log 2024-04-18T10:07:35.402 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33621.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.27499.log.gz 2024-04-18T10:07:35.403 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37503.log 2024-04-18T10:07:35.403 INFO:teuthology.orchestra.run.smithi062.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33621.log.gz 2024-04-18T10:07:35.403 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37371.log 2024-04-18T10:07:35.403 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37503.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35490.log.gz 2024-04-18T10:07:35.403 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33337.log 2024-04-18T10:07:35.403 INFO:teuthology.orchestra.run.smithi062.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37503.log.gz 2024-04-18T10:07:35.403 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33828.log 2024-04-18T10:07:35.403 INFO:teuthology.orchestra.run.smithi062.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37371.log.gz 2024-04-18T10:07:35.404 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37451.log 2024-04-18T10:07:35.404 INFO:teuthology.orchestra.run.smithi062.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33337.log.gz 2024-04-18T10:07:35.404 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36763.log 2024-04-18T10:07:35.404 INFO:teuthology.orchestra.run.smithi062.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33828.log.gz 2024-04-18T10:07:35.404 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37451.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-18T10:07:35.404 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37451.log.gz 2024-04-18T10:07:35.404 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35276.log 2024-04-18T10:07:35.404 INFO:teuthology.orchestra.run.smithi062.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36763.log.gz 2024-04-18T10:07:35.405 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34721.log 2024-04-18T10:07:35.405 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36028.log 2024-04-18T10:07:35.405 INFO:teuthology.orchestra.run.smithi062.stderr: 87.8%/var/log/ceph/ceph-client.admin.34721.log: -- replaced with /var/log/ceph/ceph-client.admin.35276.log.gz 2024-04-18T10:07:35.405 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35163.log 2024-04-18T10:07:35.405 INFO:teuthology.orchestra.run.smithi062.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34721.log.gz 2024-04-18T10:07:35.405 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36305.log 2024-04-18T10:07:35.405 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36028.log: /var/log/ceph/ceph-client.admin.35163.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35163.log.gz 2024-04-18T10:07:35.405 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36028.log.gz 2024-04-18T10:07:35.406 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37633.log 2024-04-18T10:07:35.406 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33617.log 2024-04-18T10:07:35.406 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36305.log: /var/log/ceph/ceph-client.admin.37633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33820.log 2024-04-18T10:07:35.406 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33617.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36305.log.gz 2024-04-18T10:07:35.406 INFO:teuthology.orchestra.run.smithi062.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37633.log.gz 2024-04-18T10:07:35.406 INFO:teuthology.orchestra.run.smithi062.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34251.log 2024-04-18T10:07:35.406 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33617.log.gz 2024-04-18T10:07:35.407 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35302.log 2024-04-18T10:07:35.407 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33820.log: /var/log/ceph/ceph-client.admin.34251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35976.log 2024-04-18T10:07:35.407 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35302.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34251.log.gz 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33820.log.gz 2024-04-18T10:07:35.407 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.407 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose 89.5% -- /var/log/ceph/ceph-client.admin.36435.log -- replaced with /var/log/ceph/ceph-client.admin.35302.log.gz 2024-04-18T10:07:35.407 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.407 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34381.log 2024-04-18T10:07:35.408 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35976.log: /var/log/ceph/ceph-client.admin.36435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34095.log 2024-04-18T10:07:35.408 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34381.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35976.log.gz 2024-04-18T10:07:35.408 INFO:teuthology.orchestra.run.smithi062.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36435.log.gz 2024-04-18T10:07:35.408 INFO:teuthology.orchestra.run.smithi062.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34381.log.gz 2024-04-18T10:07:35.408 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35111.log 2024-04-18T10:07:35.408 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34329.log 2024-04-18T10:07:35.409 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34095.log: /var/log/ceph/ceph-client.admin.35111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36976.log 2024-04-18T10:07:35.409 INFO:teuthology.orchestra.run.smithi062.stderr: 87.3%/var/log/ceph/ceph-client.admin.34329.log: -- replaced with /var/log/ceph/ceph-client.admin.35111.log.gz 2024-04-18T10:07:35.409 INFO:teuthology.orchestra.run.smithi062.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34095.log.gz 2024-04-18T10:07:35.409 INFO:teuthology.orchestra.run.smithi062.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34329.log.gz 2024-04-18T10:07:35.409 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36212.log 2024-04-18T10:07:35.409 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34043.log 2024-04-18T10:07:35.410 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36976.log: /var/log/ceph/ceph-client.admin.36212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33192.log 2024-04-18T10:07:35.410 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34043.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36212.log.gz 2024-04-18T10:07:35.410 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36976.log.gz 2024-04-18T10:07:35.410 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34043.log.gz 2024-04-18T10:07:35.410 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34407.log 2024-04-18T10:07:35.410 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37158.log 2024-04-18T10:07:35.410 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33192.log: /var/log/ceph/ceph-client.admin.34407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37763.log 2024-04-18T10:07:35.411 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37158.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33192.log.gz 2024-04-18T10:07:35.411 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% 91.3% -- replaced with /var/log/ceph/ceph-client.admin.34407.log.gz 2024-04-18T10:07:35.411 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-18T10:07:35.411 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37158.log.gz 2024-04-18T10:07:35.411 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37607.log 2024-04-18T10:07:35.411 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37763.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33471.log 2024-04-18T10:07:35.411 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37607.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37763.log.gz 2024-04-18T10:07:35.412 INFO:teuthology.orchestra.run.smithi062.stderr: 84.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-18T10:07:35.412 INFO:teuthology.orchestra.run.smithi062.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.37607.log.gz 2024-04-18T10:07:35.412 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35407.log 2024-04-18T10:07:35.412 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37002.log 2024-04-18T10:07:35.412 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33471.log: /var/log/ceph/ceph-client.admin.35407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33246.log 2024-04-18T10:07:35.412 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37002.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33471.log.gz 2024-04-18T10:07:35.412 INFO:teuthology.orchestra.run.smithi062.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35407.log.gz 2024-04-18T10:07:35.413 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37229.log 2024-04-18T10:07:35.413 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37002.log.gz 2024-04-18T10:07:35.413 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37319.log 2024-04-18T10:07:35.413 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33246.log: /var/log/ceph/ceph-client.admin.37229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33624.log 2024-04-18T10:07:35.413 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37319.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37229.log.gz 2024-04-18T10:07:35.413 INFO:teuthology.orchestra.run.smithi062.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33246.log.gz 2024-04-18T10:07:35.413 INFO:teuthology.orchestra.run.smithi062.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37319.log.gz 2024-04-18T10:07:35.414 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35059.log 2024-04-18T10:07:35.414 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37737.log 2024-04-18T10:07:35.414 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33624.log: /var/log/ceph/ceph-client.admin.35059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33834.log 2024-04-18T10:07:35.414 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37737.log: 88.2% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35059.log.gz -- replaced with /var/log/ceph/ceph-client.admin.33624.log.gz 2024-04-18T10:07:35.414 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.414 INFO:teuthology.orchestra.run.smithi062.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37737.log.gz 2024-04-18T10:07:35.414 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37685.log 2024-04-18T10:07:35.415 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35796.log 2024-04-18T10:07:35.415 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33834.log: /var/log/ceph/ceph-client.admin.37685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34199.log 2024-04-18T10:07:35.415 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35796.log: 88.1% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33834.log.gz 2024-04-18T10:07:35.415 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37685.log.gz 2024-04-18T10:07:35.415 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35796.log.gz 2024-04-18T10:07:35.415 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37106.log 2024-04-18T10:07:35.415 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33218.log 2024-04-18T10:07:35.416 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34199.log: /var/log/ceph/ceph-client.admin.37106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37555.log 2024-04-18T10:07:35.416 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33218.log: 88.8% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34199.log.gz 2024-04-18T10:07:35.416 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37106.log.gz 2024-04-18T10:07:35.416 INFO:teuthology.orchestra.run.smithi062.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33218.log.gz 2024-04-18T10:07:35.416 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36409.log 2024-04-18T10:07:35.416 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37867.log 2024-04-18T10:07:35.416 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37555.log: /var/log/ceph/ceph-client.admin.36409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35594.log 2024-04-18T10:07:35.417 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37867.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36409.log.gz 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37555.log.gz 2024-04-18T10:07:35.417 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.417 INFO:teuthology.orchestra.run.smithi062.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37867.log.gz 2024-04-18T10:07:35.417 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37281.log 2024-04-18T10:07:35.417 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37203.log 2024-04-18T10:07:35.417 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35594.log: /var/log/ceph/ceph-client.admin.37281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34147.log 2024-04-18T10:07:35.417 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37203.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37281.log.gz 2024-04-18T10:07:35.418 INFO:teuthology.orchestra.run.smithi062.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35594.log.gz 2024-04-18T10:07:35.418 INFO:teuthology.orchestra.run.smithi062.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37203.log.gz 2024-04-18T10:07:35.418 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34069.log 2024-04-18T10:07:35.418 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33016.log 2024-04-18T10:07:35.418 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34147.log: /var/log/ceph/ceph-client.admin.34069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33575.log 2024-04-18T10:07:35.418 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33016.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34147.log.gz 2024-04-18T10:07:35.418 INFO:teuthology.orchestra.run.smithi062.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34069.log.gz 2024-04-18T10:07:35.419 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35750.log 2024-04-18T10:07:35.419 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33016.log.gz 2024-04-18T10:07:35.419 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36898.log 2024-04-18T10:07:35.419 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33575.log: /var/log/ceph/ceph-client.admin.35750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35950.log 2024-04-18T10:07:35.419 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36898.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35750.log.gz 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33575.log.gz 2024-04-18T10:07:35.419 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.419 INFO:teuthology.orchestra.run.smithi062.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36898.log.gz 2024-04-18T10:07:35.419 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-18T10:07:35.420 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33628.log 2024-04-18T10:07:35.420 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35950.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33445.log 2024-04-18T10:07:35.420 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33628.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35950.log.gz 2024-04-18T10:07:35.420 INFO:teuthology.orchestra.run.smithi062.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33628.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35007.log 2024-04-18T10:07:35.420 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.420 INFO:teuthology.orchestra.run.smithi062.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-18T10:07:35.420 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33445.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33445.log.gz 2024-04-18T10:07:35.421 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36240.log 2024-04-18T10:07:35.421 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-18T10:07:35.421 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35007.log: /var/log/ceph/ceph-client.admin.36240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37659.log 2024-04-18T10:07:35.421 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-mds.a.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35007.log.gz 2024-04-18T10:07:35.421 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36240.log.gz 2024-04-18T10:07:35.421 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37477.log 2024-04-18T10:07:35.421 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35328.log 2024-04-18T10:07:35.422 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37659.log: /var/log/ceph/ceph-client.admin.37477.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37659.log.gz 2024-04-18T10:07:35.422 INFO:teuthology.orchestra.run.smithi062.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37477.log.gz 2024-04-18T10:07:35.422 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36737.log 2024-04-18T10:07:35.422 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35328.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35328.log.gz 2024-04-18T10:07:35.422 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35568.log 2024-04-18T10:07:35.422 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37891.log 2024-04-18T10:07:35.422 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36737.log: /var/log/ceph/ceph-client.admin.35568.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36737.log.gz 2024-04-18T10:07:35.423 INFO:teuthology.orchestra.run.smithi062.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35568.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35033.log 2024-04-18T10:07:35.423 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34121.log 2024-04-18T10:07:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35033.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37891.log.gz 2024-04-18T10:07:35.423 INFO:teuthology.orchestra.run.smithi062.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35033.log.gz 2024-04-18T10:07:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37917.log 2024-04-18T10:07:35.423 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37397.log 2024-04-18T10:07:35.423 INFO:teuthology.orchestra.run.smithi062.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34121.log.gz/var/log/ceph/ceph-client.admin.37917.log: 2024-04-18T10:07:35.424 INFO:teuthology.orchestra.run.smithi062.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37917.log.gz 2024-04-18T10:07:35.424 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34017.log 2024-04-18T10:07:35.424 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34669.log 2024-04-18T10:07:35.424 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37397.log: /var/log/ceph/ceph-client.admin.34017.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37397.log.gz 2024-04-18T10:07:35.424 INFO:teuthology.orchestra.run.smithi062.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34017.log.gz 2024-04-18T10:07:35.424 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33626.log 2024-04-18T10:07:35.424 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33830.log 2024-04-18T10:07:35.425 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34669.log: /var/log/ceph/ceph-client.admin.33626.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34669.log.gz 2024-04-18T10:07:35.425 INFO:teuthology.orchestra.run.smithi062.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33626.log.gz 2024-04-18T10:07:35.425 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36106.log 2024-04-18T10:07:35.425 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33523.log 2024-04-18T10:07:35.425 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33830.log: /var/log/ceph/ceph-client.admin.36106.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33830.log.gz 2024-04-18T10:07:35.425 INFO:teuthology.orchestra.run.smithi062.stderr: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.36106.log.gz 2024-04-18T10:07:35.425 INFO:teuthology.orchestra.run.smithi062.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35724.log 2024-04-18T10:07:35.425 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33523.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33523.log.gz 2024-04-18T10:07:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33363.log 2024-04-18T10:07:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35085.log 2024-04-18T10:07:35.426 INFO:teuthology.orchestra.run.smithi062.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35724.log.gz 2024-04-18T10:07:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33363.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33363.log.gz 2024-04-18T10:07:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37255.log 2024-04-18T10:07:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34929.log 2024-04-18T10:07:35.426 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35085.log: /var/log/ceph/ceph-client.admin.37255.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35085.log.gz 2024-04-18T10:07:35.426 INFO:teuthology.orchestra.run.smithi062.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37255.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34851.log 2024-04-18T10:07:35.427 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.427 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34929.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34929.log.gz 2024-04-18T10:07:35.427 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35823.log 2024-04-18T10:07:35.427 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37132.log 2024-04-18T10:07:35.427 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34851.log: /var/log/ceph/ceph-client.admin.35823.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34851.log.gz 2024-04-18T10:07:35.427 INFO:teuthology.orchestra.run.smithi062.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35823.log.gz 2024-04-18T10:07:35.427 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36789.log 2024-04-18T10:07:35.428 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22731.log 2024-04-18T10:07:35.428 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37132.log: /var/log/ceph/ceph-client.admin.36789.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37132.log.gz 2024-04-18T10:07:35.428 INFO:teuthology.orchestra.run.smithi062.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36789.log.gz 2024-04-18T10:07:35.428 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35222.log 2024-04-18T10:07:35.428 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36054.log 2024-04-18T10:07:35.428 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph.tmp-client.admin.22731.log: /var/log/ceph/ceph-client.admin.35222.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22731.log.gz 2024-04-18T10:07:35.428 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33835.log 2024-04-18T10:07:35.428 INFO:teuthology.orchestra.run.smithi062.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35222.log.gz 2024-04-18T10:07:35.429 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36054.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36054.log.gz 2024-04-18T10:07:35.429 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34799.log 2024-04-18T10:07:35.429 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33620.log 2024-04-18T10:07:35.429 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33835.log: /var/log/ceph/ceph-client.admin.34799.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33835.log.gz 2024-04-18T10:07:35.429 INFO:teuthology.orchestra.run.smithi062.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34799.log.gz 2024-04-18T10:07:35.429 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34695.log 2024-04-18T10:07:35.429 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34277.log 2024-04-18T10:07:35.429 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33620.log: /var/log/ceph/ceph-client.admin.34695.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34695.log.gz 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33620.log.gz 2024-04-18T10:07:35.430 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.430 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36002.log 2024-04-18T10:07:35.430 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35849.log 2024-04-18T10:07:35.430 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34277.log: /var/log/ceph/ceph-client.admin.36002.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34277.log.gz 2024-04-18T10:07:35.430 INFO:teuthology.orchestra.run.smithi062.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37789.log 2024-04-18T10:07:35.430 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36002.log.gz 2024-04-18T10:07:35.430 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35849.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35849.log.gz 2024-04-18T10:07:35.430 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37529.log 2024-04-18T10:07:35.431 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35189.log 2024-04-18T10:07:35.431 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.37789.log: /var/log/ceph/ceph-client.admin.37529.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37789.log.gz 2024-04-18T10:07:35.431 INFO:teuthology.orchestra.run.smithi062.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37529.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36540.log 2024-04-18T10:07:35.431 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.431 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35189.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35189.log.gz 2024-04-18T10:07:35.431 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35646.log 2024-04-18T10:07:35.431 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33166.log 2024-04-18T10:07:35.432 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.36540.log: /var/log/ceph/ceph-client.admin.35646.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35646.log.gz 2024-04-18T10:07:35.432 INFO:teuthology.orchestra.run.smithi062.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.36540.log.gz 2024-04-18T10:07:35.432 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37841.log 2024-04-18T10:07:35.432 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35459.log 2024-04-18T10:07:35.432 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.33166.log: /var/log/ceph/ceph-client.admin.37841.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33166.log.gz 2024-04-18T10:07:35.432 INFO:teuthology.orchestra.run.smithi062.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37841.log.gz 2024-04-18T10:07:35.432 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35620.log 2024-04-18T10:07:35.432 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35459.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35459.log.gz 2024-04-18T10:07:35.433 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.35620.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35620.log.gz 2024-04-18T10:07:35.433 INFO:teuthology.orchestra.run.smithi062.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-18T10:07:35.434 INFO:teuthology.orchestra.run.smithi062.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-18T10:07:35.434 INFO:teuthology.orchestra.run.smithi062.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-18T10:07:35.434 INFO:teuthology.orchestra.run.smithi145.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-18T10:07:35.434 INFO:teuthology.orchestra.run.smithi145.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-18T10:07:35.435 INFO:teuthology.orchestra.run.smithi062.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-18T10:07:35.441 INFO:teuthology.orchestra.run.smithi062.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-18T10:07:35.450 INFO:teuthology.orchestra.run.smithi145.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-18T10:07:35.459 INFO:teuthology.orchestra.run.smithi145.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-18T10:07:35.461 INFO:teuthology.orchestra.run.smithi145.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-18T10:07:35.479 INFO:teuthology.orchestra.run.smithi062.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-18T10:07:35.480 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-04-18T10:07:35.480 INFO:teuthology.orchestra.run.smithi062.stderr:real 0m0.219s 2024-04-18T10:07:35.481 INFO:teuthology.orchestra.run.smithi062.stderr:user 0m0.844s 2024-04-18T10:07:35.481 INFO:teuthology.orchestra.run.smithi062.stderr:sys 0m0.105s 2024-04-18T10:07:35.497 INFO:teuthology.orchestra.run.smithi145.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-18T10:07:35.498 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-04-18T10:07:35.499 INFO:teuthology.orchestra.run.smithi145.stderr:real 0m0.181s 2024-04-18T10:07:35.499 INFO:teuthology.orchestra.run.smithi145.stderr:user 0m0.681s 2024-04-18T10:07:35.499 INFO:teuthology.orchestra.run.smithi145.stderr:sys 0m0.042s 2024-04-18T10:07:35.499 INFO:tasks.ceph:Archiving logs... 2024-04-18T10:07:35.500 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928/remote/smithi062/log 2024-04-18T10:07:35.500 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-18T10:07:35.759 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928/remote/smithi145/log 2024-04-18T10:07:35.760 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-18T10:07:35.856 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-18T10:07:35.877 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-18T10:07:35.878 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-18T10:07:35.880 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-18T10:07:35.955 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-18T10:07:35.964 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-18T10:07:35.965 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-18T10:07:35.967 DEBUG:teuthology.orchestra.run.smithi145:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-18T10:07:35.977 INFO:teuthology.orchestra.run.smithi062.stdout: remote refid st t when poll reach delay offset jitter 2024-04-18T10:07:35.978 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================== 2024-04-18T10:07:35.978 INFO:teuthology.orchestra.run.smithi062.stdout:+hv01.front.sepi 67.205.162.81 3 u - 64 77 0.091 -1.129 0.800 2024-04-18T10:07:35.978 INFO:teuthology.orchestra.run.smithi062.stdout:+hv02.front.sepi 63.231.80.2 3 u 64 64 37 0.100 -0.147 0.551 2024-04-18T10:07:35.978 INFO:teuthology.orchestra.run.smithi062.stdout:*hv03.front.sepi 74.6.168.72 3 u 64 64 37 0.096 -0.672 0.524 2024-04-18T10:07:35.978 INFO:teuthology.orchestra.run.smithi062.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T10:07:35.981 INFO:teuthology.orchestra.run.smithi145.stdout: remote refid st t when poll reach delay offset jitter 2024-04-18T10:07:35.981 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================== 2024-04-18T10:07:35.981 INFO:teuthology.orchestra.run.smithi145.stdout:+hv01.front.sepi 67.205.162.81 3 u 59 64 37 0.103 -2.465 0.744 2024-04-18T10:07:35.981 INFO:teuthology.orchestra.run.smithi145.stdout:*hv02.front.sepi 63.231.80.2 3 u 1 64 77 0.103 -0.121 1.146 2024-04-18T10:07:35.982 INFO:teuthology.orchestra.run.smithi145.stdout:+hv03.front.sepi 74.6.168.72 3 u 61 64 37 0.138 -1.518 0.791 2024-04-18T10:07:35.982 INFO:teuthology.orchestra.run.smithi145.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-18T10:07:35.983 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-18T10:07:35.993 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-18T10:07:35.993 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-18T10:07:36.036 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-18T10:07:36.069 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-18T10:07:36.103 INFO:teuthology.task.internal:Duration was 768.240791 seconds 2024-04-18T10:07:36.103 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-18T10:07:36.136 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-18T10:07:36.137 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-18T10:07:36.139 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-18T10:07:36.177 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-18T10:07:36.178 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2024-04-18T10:07:36.178 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-18T10:07:36.229 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi145.front.sepia.ceph.com 2024-04-18T10:07:36.229 DEBUG:teuthology.orchestra.run.smithi145:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-18T10:07:36.239 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-18T10:07:36.239 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-18T10:07:36.273 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-18T10:07:36.306 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-18T10:07:36.306 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-18T10:07:36.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-18T10:07:36.406 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-18T10:07:36.416 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-18T10:07:36.416 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-18T10:07:36.429 DEBUG:teuthology.orchestra.run.smithi145:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-18T10:07:36.439 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-18T10:07:36.449 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-18T10:07:36.473 DEBUG:teuthology.orchestra.run.smithi145:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-18T10:07:36.481 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2024-04-18T10:07:36.488 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = core 2024-04-18T10:07:36.517 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-18T10:07:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T10:07:36.540 DEBUG:teuthology.orchestra.run.smithi145:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-18T10:07:36.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-18T10:07:36.564 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-18T10:07:36.574 INFO:teuthology.task.internal:Transferring archived files... 2024-04-18T10:07:36.574 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928/remote/smithi062 2024-04-18T10:07:36.575 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-18T10:07:36.605 DEBUG:teuthology.misc:Transferring archived files from smithi145:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928/remote/smithi145 2024-04-18T10:07:36.605 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-18T10:07:36.628 INFO:teuthology.task.internal:Removing archive directory... 2024-04-18T10:07:36.628 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-18T10:07:36.645 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-18T10:07:36.673 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-18T10:07:36.684 INFO:teuthology.task.internal:Not uploading archives. 2024-04-18T10:07:36.684 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-18T10:07:36.712 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-18T10:07:36.712 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-18T10:07:36.714 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-18T10:07:36.718 INFO:teuthology.orchestra.run.smithi062.stdout: 658644 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 18 10:07 /home/ubuntu/cephtest 2024-04-18T10:07:36.721 INFO:teuthology.orchestra.run.smithi145.stdout: 658633 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 18 10:07 /home/ubuntu/cephtest 2024-04-18T10:07:36.722 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-18T10:07:36.746 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-18T10:07:36.813 INFO:teuthology.nuke:Checking targets against current locks 2024-04-18T10:07:36.838 DEBUG:teuthology.nuke:shortname: smithi062 2024-04-18T10:07:36.838 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-18T10:07:36.859 DEBUG:teuthology.nuke:shortname: smithi145 2024-04-18T10:07:36.860 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-18T10:07:36.863 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 09:46:24.217210', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPPVfMD4DHC87XmWZt3TofmRlDjTt43T/RfV4ZFvgP2nfzE+gBYIP68S+VzT0UuVvprZV3CcxGQOBax7qpzfhkI='} 2024-04-18T10:07:36.885 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-17_20:24:14-fs-main-distro-default-smithi/7660928', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-18 09:46:24.215129', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN0dbhDmL1EjXSQwVknmV/Y3yOKu6mDKjuLciUf2po3nF29NJv8cXAwqEelT3Q3MjvvxUvBSEnsiWju2Ip+rIVs='} 2024-04-18T10:07:36.913 INFO:teuthology.orchestra.console.smithi062:Power off 2024-04-18T10:07:36.914 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-18T10:07:36.945 INFO:teuthology.orchestra.console.smithi145:Power off 2024-04-18T10:07:36.945 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-18T10:07:36.955 DEBUG:teuthology.orchestra.console.smithi062:power off output: Chassis Power Control: Down/Off 2024-04-18T10:07:36.956 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T10:07:36.972 DEBUG:teuthology.orchestra.console.smithi145:power off output: Chassis Power Control: Down/Off 2024-04-18T10:07:36.973 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T10:07:36.985 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-04-18T10:07:36.998 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-04-18T10:07:40.986 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T10:07:40.999 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T10:07:41.097 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-04-18T10:07:41.110 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-04-18T10:07:45.099 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T10:07:45.112 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T10:07:45.211 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-04-18T10:07:45.223 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-04-18T10:07:49.216 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T10:07:49.226 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-18T10:07:49.328 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is off 2024-04-18T10:07:49.338 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is off 2024-04-18T10:07:49.429 INFO:teuthology.orchestra.console.smithi062:Power off completed 2024-04-18T10:07:49.439 INFO:teuthology.orchestra.console.smithi145:Power off completed 2024-04-18T10:07:49.547 INFO:teuthology.run:Summary data: description: fs/shell/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1-mds-1-client-coloc conf/{client mds mgr mon osd} distro/ubuntu_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cephfs-shell} duration: 768.2407913208008 failure_reason: 'Test failure: test_cd_with_args (tasks.cephfs.test_cephfs_shell.TestCD)' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1266d81a082e46e284e8cea1bbb9f61d status: fail success: false 2024-04-18T10:07:49.547 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-18T10:07:49.637 INFO:teuthology.run:FAIL